GPU prices as of 6 Nov 2026
Compute Census

AI Inference vs Training: Hardware, Power and Cost

Guide · Training and inference · updated · 22 min read

AI inference vs training compared as infrastructure: GPU memory and bandwidth, cluster size, rack power, location and what each job costs on rented GPUs.

By , founder of the publisher, The Dream View Co., Ltd. · first published .

Cover: AI Inference vs Training: Hardware, Power and Cost — $3.90, median on-demand H100 rental per GPU-hour, Oct 2026

The short answer

Training and inference are two different jobs that happen to run on the same chips. Training is the one-off, very large job that produces a model, run for weeks on thousands of tightly connected GPUs. Inference is the service that uses the finished model, run for as long as the product exists, on as many or as few GPUs as demand needs. The difference shows up as infrastructure: how many GPUs sit together, how fast they talk to each other, how much memory each one carries, where the building is, and how the bill is paid.

Training is a project with a start and an end. It is bound by compute and by the network between GPUs, so it wants many identical GPUs on one fabric, running at high utilisation for a fixed period. Inference is a utility. Its limit is usually memory and latency rather than raw compute, so it wants enough memory per GPU to hold the model and its working state, and it wants capacity near users that can grow and shrink with traffic.

The prices in our index reflect the overlap. At the Compute Census snapshot of 6 Nov 2026, the median on-demand price of an H100 is $3.90 per GPU-hour, and the H200, with more memory, is $4.80. The newer B200 is $6.90. Which one is the better value depends on which of the two jobs you are buying it for, and the rest of this guide explains why.

$3.90H100 median, on-demand, per GPU-hourindex of 6 Nov 2026
$1.30floorcheapest accepted on-demand offer
$12.29ceiling9.5× the floor
$3.2612-month reserved−16% vs median

We cover the two workloads in turn, then the hardware each one rewards, the cluster sizes and power they imply, the effect on location, and what each costs on rented GPUs. Every external figure is dated and listed on our Sources page, and every rental figure is read from our index at build time. For the arithmetic that turns a GPU-hour price into a price per token, see our guide to AI inference cost.

Two jobs that look alike from far away

What training does

A neural network starts as a set of random numbers. Training adjusts those numbers over many passes through a large body of data until the model produces useful output. Each pass reads data, computes the model’s answer, measures the error and updates every parameter. The work is repeated trillions of times, and the parameters have to stay in step across every GPU taking part.

That last point is what makes training an infrastructure problem and not only an arithmetic one. When a model is too large for one GPU, or too slow on one, the job is split across many, and the GPUs exchange results at every step. A slow link, a failed node or an unbalanced load stalls everyone. Meta’s engineering team said that an unoptimised large cluster of its own ran its communication anywhere from about 10 percent to 90 percent efficient, and returned to above 90 percent only after a full-system optimisation. A training cluster is as fast as its network and as reliable as its worst node.

What inference does

Inference is the finished model answering a request. A user sends a prompt, the model reads it, and the model writes an answer one token at a time. No parameters change. The same weights are read again and again, for every request, from every user, for as long as the service is live.

NVIDIA’s engineers describe two phases. In prefill, the model processes the input tokens in parallel, which saturates the GPU. In decode, output tokens are generated one at a time, and NVIDIA calls this a memory-bound operation, because moving data from memory takes longer than the arithmetic. Prefill is a compute job and decode is a memory job, and a single request needs both.

Why the difference matters for a buyer

The two jobs pay for different things. A training buyer pays for a large block of GPU-hours that must all be available together, and the total is known in advance, because the run has a planned length. An inference buyer pays for capacity that follows demand. If a product goes quiet, the training bill stops at the end of the run, but the inference bill carries on as long as the capacity is reserved. The buyers’ mistakes follow from that. Training buyers over-estimate how much of the cluster they will actually use, and inference buyers under-estimate how long the service will live.

What each job asks of the hardware

Memory: capacity and bandwidth

Inference leans on memory in two ways. The first is capacity. The weights of the model have to fit, with room left for the key-value cache, which holds the attention state of every request in flight. NVIDIA’s article puts the cache for a 7-billion-parameter Llama 2 model at about 2 GB for a single sequence of 4,096 tokens in 16-bit precision. That is for one request on a small model. A service handling many requests at once on a larger model needs many times that, and the cache competes with the weights for the same memory.

The second is bandwidth. Because decode reads the weights from memory for every token generated, the speed at which memory can be read sets how fast tokens come out. NVIDIA’s own pages make the case by what they advertise. The H100 is listed with 80 GB of memory at 3.35 TB/s in the SXM version. The H200 raises that to 141 GB at 4.8 TB/s, and NVIDIA claims up to twice the inference speed of the H100 on large language models such as Llama 2, with 1.9 times for Llama 2 70B and 1.6 times for GPT-3 175B under the footnoted test conditions. The page makes no training speed claim for the H200. The change the vendor chose to advertise is a memory change, and the claim it chose is an inference claim.

Training needs memory too, and more of it than inference does for the same model, because it holds gradients and optimiser state besides the weights. But a training job can spread that state across many GPUs, so it is rarely limited by one card. It is limited by how fast the cards can synchronise.

The interconnect

The link between GPUs is the most important difference. A training run exchanges results among all its GPUs at every step, so the network sets a ceiling on speed. That is why training clusters are built around a dedicated fabric. Meta’s two 24,576-GPU clusters each connect 400 Gbps endpoints, one over Ethernet with RDMA (RoCE) built on Arista switches and the other over NVIDIA’s Quantum2 InfiniBand.

Inside a server, the link is faster still. NVIDIA’s GB200 NVL72 page lists 1.8 TB/s of NVLink bandwidth per GPU and 130 TB/s across the 72-GPU rack, which makes the rack one NVLink domain. NVIDIA’s claims for that design are about both jobs: 30 times the real-time LLM inference performance of the H100 and 4 times the training performance, at test conditions listed in the footnotes. Those conditions include a 1.8-trillion-parameter mixture-of-experts model trained on 4,096 H100 GPUs against 456 NVL72 systems, and a comparison of 32,768 H100 GPUs against NVL72 for inference. They are vendor projections at a specific scale. We quote them for the ratio between the two: the stated gain for inference is much larger than for training, which is consistent with the rack’s design goal, a very large memory and interconnect domain that serves one very large model.

An inference service on a model that fits one GPU, or one eight-GPU server, needs far less. Its requests are independent, so the GPUs do not have to exchange results at every step. The network matters at the edge of the service, where requests arrive, and not between the GPUs.

Compute precision

Both jobs use the same arithmetic units but not always at the same precision. Training usually runs at higher precision to keep the numbers stable, while inference can often use lower precision to fit more of the model in memory and to read it faster. NVIDIA’s Rubin CPX press release, dated 9 September 2025, shows where this leads. The company described it as a GPU purpose-built for massive-context processing, with 128 GB of GDDR7 memory and up to 30 petaflops of NVFP4 compute, and said it was expected at the end of 2026. GDDR7 is the memory of graphics cards, not the stacked HBM of the flagship parts. The design trades bandwidth for cost on a part meant for one phase of inference. A chip for only one phase of one job is a sign that the market sees the jobs as different enough to build different silicon.

Table 1GPU memory and rental price in the index, USD per GPU-hour

ModelMedian12-mo reservedMemoryMedian per GB of memoryMedian × 730 hFloorCeilingProviders
H100$3.90$3.2680 GB$0.049$2,847$1.30$12.2911
H200$4.80$3.40141 GB$0.034$3,504$2.09$13.789
B200$6.90not published180 GB$0.038$5,037$3.69$16.1112
MI300X$2.99$1.91192 GB$0.016$2,183$1.71$12.007
L40S$1.53$1.0948 GB$0.032$1,117$0.38$3.7710

USD per GPU-hour, on-demand unless stated. Compute Census GPU price index of 6 Nov 2026.

Table 1 puts the memory side by side. The per-gigabyte column is the one to read first for inference on large models, where memory decides how many GPUs a job needs. It is less useful for training, where compute and the network count for more. A GPU that costs more per hour and less per gigabyte can be the cheaper way to serve a model that does not fit elsewhere.

How big is a training cluster, and how big is an inference fleet

Training: thousands of GPUs, one purpose

The largest training clusters are very large. Meta’s March 2024 engineering post describes two clusters of 24,576 H100 GPUs each, built to train models such as Llama 3, and succeeding a research cluster of 16,000 A100 GPUs. The same post says the company aimed to reach 350,000 H100 GPUs by the end of 2024, and compute equal to nearly 600,000 H100s. Those are plans and capacities, not a statement of how many GPUs any single run used.

The model card for Llama 3.1 gives the other side: the total effort. Meta’s model card for Llama 3.1 lists 30.84 million GPU hours on H100-80GB hardware for the 405B model, at 700 W per GPU, and 39.3 million GPU hours for the three sizes together. In detail, Meta lists 1.46 million GPU hours for the 8B model, 7.0 million for the 70B model and 30.84 million for the 405B model, which makes 39.3 million GPU hours in total on H100-80GB hardware rated at 700 W per GPU. Meta’s card says the location-based emissions of the training were 11,390 tons of CO2 equivalent, and the market-based figure was zero.

A GPU hour is the unit that links the cluster to the bill. Thirty million GPU hours spread over a 24,576-GPU cluster is about 1,250 hours per GPU, which is a little over 50 days if every GPU were busy the whole time. The model card does not say how many GPUs a particular run used or how long it lasted, so we do not compute a duration from it. The point is the scale: one of the large models took a cluster’s entire year-sized capacity, a quantity that no individual customer of a cloud can take without notice.

Training cost is mostly the hardware

Epoch AI’s 2023 study of training costs is the best-known published method. It estimated the dollar cost of 124 systems from 2009 to 2022, and found that the cost of training grew at about 0.49 orders of magnitude a year across all systems and about 0.2 a year for large-scale systems since late 2015. Its main estimate covers hardware only: training compute divided by the compute a dollar buys, using an assumed 35 percent utilisation and a two-year replacement time. It excludes cloud rental prices because they include the provider’s margin, and puts energy at about 20 percent of the hardware cost in an appendix, using GPT-3 as the example. The author says his own figures are likely underestimates, and notes other sources with higher numbers for the same systems.

The study is from January 2023, so it says nothing about H100-era runs. We cite it for its method and its assumptions, which are useful to any buyer: a training budget is dominated by the hardware time, utilisation is the main lever, and energy is a smaller line. Our rental index lets you try the same arithmetic with a market price. Take a published GPU-hour count, such as the 30.84 million for Llama 3.1 405B, and multiply by the H100 price in the index. That is a rental-price cost for the run at the index’s median. It is not what Meta paid, since Meta owns its hardware, and it is not what a customer would be charged for a block of that size, since large blocks are negotiated. It is a scale check, nothing more.

Inference: many small units, scaled by traffic

An inference fleet is a different shape. It is made of many servers, each running a copy of the model, behind a load balancer. If a model fits on one eight-GPU server, the fleet scales by adding servers. If it does not, each copy spans several servers, and the unit that scales is the copy.

Meta does not break out what share of its fleet serves inference. Its engineering post says the company handles “hundreds of trillions” of AI model executions per day, and describes the two 24,576-GPU clusters as training infrastructure. No published figure we found gives a global split of GPUs between training and inference, so we do not give one, and we are wary of anyone who does without a source.

The reason a fleet is harder to size than a cluster is that the demand is not known in advance. Google’s paper on the energy per prompt, summarised on its Cloud blog on 21 August 2025, includes in its full measurement the energy of idle machines, which are provisioned chips kept ready for availability, failover and traffic spikes. That is a cost training does not have. A training cluster runs flat out until the job is done. An inference fleet has to hold spare capacity, and the spare capacity is paid for.

Power and cooling: the rack decides

The server, in numbers

A DGX H100 or H200 is the reference eight-GPU server for both jobs. NVIDIA’s user guide gives it 640 GB of GPU memory in the H100 version and 1,128 GB in the H200 version, a maximum input of 10.2 kW from six 3.3 kW power supplies, a height of 8U and a weight of up to 287.6 pounds. For networking it carries eight ConnectX-7 InfiniBand cards at up to 400 Gbps for the compute fabric and two dual-port ConnectX-7 Ethernet cards for storage and management.

Those numbers are the same whether the server trains or serves. The difference is how many of them sit together and what connects them. A training pod is a few hundred servers on a non-blocking fabric, drawing megawatts at one address. An inference site can be one rack in a colocation hall or a hundred, and the servers do not have to be next to each other.

The rack-scale step

At the top of the range, the rack is the product. NVIDIA’s GB200 NVL72 page describes a fully liquid-cooled rack with 13.4 TB of HBM3e memory at 576 TB/s, joined by 130 TB/s of NVLink, and says liquid cooling reduces the floor space needed and the data center’s energy use and water consumption. We covered the electrical numbers in our guide to the NVIDIA B200 price; the point here is that a liquid-cooled rack is a facility decision, not a purchase order. A hall without chilled liquid and the power to match cannot take one.

A rack of that class suits very large training runs and very large model serving, because both want a single large memory and interconnect domain. It is overkill for the many inference deployments of models that fit on one server. For those, an air-cooled eight-GPU server at around 10 kW is the product, and it can go where the power already is.

Energy per answer

For inference the question is not the rack but the answer. Google’s August 2025 paper reports that the median Gemini Apps text prompt used 0.24 Wh, 0.03 gCO2e and 0.26 mL of water, measured over May 2025 with a comprehensive method that counts the full system, idle machines, host CPU and memory, and the data center’s overhead. A narrower count of active chips alone gave 0.10 Wh, which Google calls optimistic. It reports that the median energy per prompt fell 33-fold and the carbon footprint 44-fold between May 2024 and May 2025. Google says its figures are a point-in-time analysis, do not represent every Gemini prompt and have not been independently verified.

Two readings of that number are useful, and one is not. It is useful to see that a single answer costs a small amount of energy, and that the measured cost depends heavily on what is included: counting the idle machines and the building more than doubled the figure. It is useful to see that the cost per answer can fall quickly as models and serving software improve. It is not useful to multiply it by a guess of daily prompts, because the volume is not published, and a total built on a guess is a guess.

Energy per answer is small; energy over a year of service is not. Training energy is a one-time lump, and inference energy is a stream that lasts as long as the product does. Neither is larger in general. Which one dominates depends on how often the model is used compared with how large it was to build, and for the most popular services the stream is the larger of the two over the model’s life. We have found no published split by workload for the world’s fleet, and we will not guess one.

Location and latency

Training can go where the power is

A training run does not care where it happens. Its users are the engineers who launched it, the data can be moved in advance and nobody waits for an answer. That makes the location a power and cost decision. A training cluster goes where a large block of power is available, where land is cheap and where the grid can support it, even if the site is far from any user.

That is the logic of the large campuses in our census. As of 15 Nov 2026, the census lists 665 AI campuses in 56 countries. 27 of them are rated at a gigawatt or more, and the median campus that publishes a figure is rated at 70 MW. Campuses of that size are built for big training runs and the large clusters that serve the biggest models, because only they need that much capacity in one place.

Table 2AI campuses in the census by published capacity class

Capacity classCampusesWith published MWPublished MW
1 GW or more272741,730
500–999 MW181811,386
100–499 MW15615633,370
Under 100 MW29529511,380
Not published16900

Compute Census AI campus census, records to 15 Nov 2026. MW summed only where a source publishes a figure. Full list: AI data center census.

Inference goes where the users are

Inference sits in the path of a user’s request. A person waiting for an answer notices delay, and a service that calls a model for every step of an agent loop notices it more. Inference capacity is therefore placed in many regions, close to users and close to the data it reads. That makes a smaller site in a well-connected metro more useful than a distant giant one.

The practical consequence for a buyer is that the inference footprint is a list of regions and the training footprint is a single site. A company can train in one country and serve in twenty. It can also train on one provider and serve on another, because the weights are a file that moves.

Data and regulation

Two constraints cut across both. The data that trains a model may have to stay within a jurisdiction, and the data a user sends to an inference service may too. The first limits where a training run can go, the second where a service can be served from. Neither is a technical matter, and both belong in the first conversation with a provider, not the last. We do not summarise the rules here, because they differ by country and change.

What each costs on rented GPUs

The index

Renting moves the infrastructure question into a price list. At the snapshot of 6 Nov 2026, the index counts 11 providers with a published H100 price and 15 price points. The floor is $1.30, the median $3.90 and the ceiling $12.29. The reserved rate, where one is published, is $3.26.

Table 3H100 prices by provider, USD per GPU-hour

ProviderOfferUSD / GPU-hrAs of
Lium.ioMarketplace$1.302026-09-21
BeamOn-demand$1.832026-10-20
CoreWeaveSpot$2.462026-10-20
RunpodOn-demand$2.692026-10-16
CivoReserved, 12 mo$2.692026-10-20
CivoOn-demand$2.992026-10-20
DigitalOceanReserved, 12 mo$3.262026-09-28
RunpodOn-demand$3.992026-10-16
LambdaOn-demand$3.99–$4.292026-10-16
DigitalOceanOn-demand$4.412026-10-20
NebiusOn-demand$4.502026-10-16
CoreWeaveOn-demand$6.162026-10-20
AWSOn-demand~$6.882026-10-16
Google CloudOn-demand~$11.002026-10-16
Microsoft AzureOn-demand$12.292026-10-16

Price points recorded by the Compute Census index for the H100; third-party index rows are context, not providers. Full table: H100 rental price.

The table has a structure that helps with the two jobs. A training buyer needs many GPUs together on one fabric, which the on-demand marketplaces at the bottom of the table often cannot supply in one block. An inference buyer needs a handful of GPUs for as long as the service lives, which a marketplace or a neocloud can supply, but which the reserved rate fits better than an hourly one if the service is steady.

The history

Figure 1H100: median on-demand rental price and published range, USD per GPU-hour, 2023-06-01 to 2026-11-06

$0$2$4$6$8$10$12$14$16202420252026
Line = median; band = floor to ceiling; red square = latest point. Dashed = before 1 Aug 2026, seeded from published indices. From the Compute Census GPU price index.

Read the chart before you read any percentage. Against 1 Oct 2025, the median has moved +100%. The first median we recorded, on 1 Jun 2023, was $8.00. The price of the workhorse GPU is the reference for both jobs, because it is the part most clouds stock in volume.

Which commitment fits which job

Training is the easier job to buy. The length is known, the size is known, and the GPU-hours can be quoted as a block. A buyer can ask for a fixed number of GPUs for a fixed number of weeks and compare offers on that basis. The price per GPU-hour then depends on the size of the block, the term, the fabric and the start date, and not on the list price alone.

Inference is harder. The demand curve is not known, so the buyer has three options. The first is on-demand capacity, paid by the hour, which costs more per hour and leaves nothing to waste. The second is reserved capacity for the base load, which costs less per hour but is paid whether it is used or not. The third is a managed endpoint, billed per token, where the provider carries the capacity risk and charges for it. The first two are GPU rental, which this index prices. The third is a different product, and our inference cost guide compares it with renting.

A common and sensible pattern is a reserved base for steady traffic with on-demand capacity for peaks. For a training buyer the equivalent pattern is a reserved block for the planned run and on-demand capacity for experiments before it.

Utilisation is the number that decides

The price per GPU-hour is the same whether the GPU is busy or not, so the cost of the work depends on utilisation. A training cluster running at 35 percent utilisation, the figure Epoch AI used as its assumption, costs nearly three times as much per unit of work as one running flat out. An inference fleet that holds spare capacity for spikes costs more per answer than one kept busy, and Google’s inclusion of idle machines in its per-prompt figure shows how large that effect can be.

That is why the buyer’s first question for either job is not the price but the expected utilisation. A cheap GPU at low utilisation costs more than a dear one at high utilisation. The index gives the price, and only the buyer’s own workload gives the utilisation.

Which GPUs suit which job

The matches below are guidance, not rules. The same chips serve both jobs, and which is best depends on the model, the batch size, the context length and the software. Use the memory and price columns of Table 1 for the comparison.

Large training runs

For training from scratch, the buyer wants many identical GPUs on one fabric, the highest compute per GPU and the fastest interconnect. The H100 and its successors are the parts the published large runs used. The Llama 3.1 card lists H100-80GB hardware, and Meta’s clusters used H100s. A100s are the older part with a large installed base, and they cost less per hour, but they take more hours for the same job. Our guide to H100 vs A100 works through when the cheaper part is the cheaper job.

Fine-tuning

Fine-tuning adapts an existing model with a small data set. It needs far fewer GPU hours than pre-training, often on one server, and it does not need a large fabric. A buyer can use whichever part has enough memory for the model and the training state. For many projects that is a single eight-GPU node, rented by the day.

Serving large models

For a large model, the buyer wants memory first. The H200 is the case NVIDIA makes: 141 GB at 4.8 TB/s, with the claims of up to twice the inference speed of the H100 on large language models. Whether the premium is worth it depends on how many H100s the model would otherwise need and on the batch size. Our guide to H100 vs H200 sets out that comparison, and the AMD MI300X, with its large memory, is the other candidate in Table 1. The per-gigabyte column puts a number on the memory side.

Serving small and medium models

A model that fits in the memory of one GPU does not need an expensive part. The L40S and other lower-priced GPUs in the index serve such models, and their lower hourly price can beat a flagship on cost per answer even if the flagship is faster. The right test is price divided by throughput, measured on your own model.

Questions to settle before you buy

For a training run

The first question is how many GPU hours the job needs, with a margin for restarts. The second is how many GPUs you can use at once, which depends on the fabric. The third is how failures are handled: at cluster scale, nodes fail during a run, and a provider’s replacement time becomes part of your schedule. Meta’s post notes that checkpointing across thousands of ranks takes hundreds of milliseconds and that start-up times for its process groups fell from hours to minutes, which shows how much engineering a large run needs from the software side as well as the hardware.

The fourth question is the start date. Capacity for a large block is not always available when you want it, and our guide to the GPU shortage explains how the market has handled that.

For an inference service

The first question is the model’s memory footprint, with the cache for the number of requests you expect at once. The second is the latency target, because it sets how many requests a GPU can batch. The third is where users are. The fourth is how steady the traffic is, which decides the split between reserved and on-demand capacity.

For either

Ask what is included in the hourly price: network, storage, support and egress. Ask what the exit terms are. Ask where the data sits. These questions are the same for both jobs, and our guide to renting a GPU covers them in more detail.

Five mistakes when comparing the two

Treating the GPU-hour price as the job cost

A GPU-hour price is an input. The job cost is the price times the hours, and the hours depend on utilisation, failures and software. Two providers with the same price can differ by a wide margin in the cost of a finished job.

Buying for training and using for inference

A cluster bought for one training run may sit idle afterwards. If the plan is to serve the model on the same hardware, check that the fabric you paid for is not the largest part of the bill. An inference fleet does not need it.

Buying inference capacity on the strength of a training benchmark

A benchmark that measures training throughput says little about decode speed, which depends on memory bandwidth and batch size. Use the benchmark that matches the job, and the model, batch size and context length you will really run.

Ignoring idle capacity

Spare capacity for spikes is a cost, and so is capacity bought for a launch that did not grow. Google counts its idle machines in its energy figure for that reason. A buyer should count them in the budget.

Assuming the answer is permanent

The right hardware for a job changes as models and software change. Google reports a 33-fold fall in median energy per prompt in a year, mostly from changes in models and serving, not a change in the physics of a building. A cluster sized for last year’s software may be too large or too small for this year’s. Prefer terms that let you change your mind.

How to source capacity for each job

Say which job it is

When you ask a provider for capacity, start with the job. A training request needs the number of GPUs, the length of the run, the fabric, the start date and the checkpoint and storage needs. An inference request needs the model and its memory footprint, the target latency, the regions, the expected traffic and how it varies.

Compare like with like

Normalise every quote to one GPU-hour, and note what each includes. A node price divided by eight is a per-GPU price. A reserved rate paid for a year is not comparable to an on-demand rate paid for a week without noting the term. Our price index does that normalisation for the published prices, and the model page for each GPU lists every provider’s quote with its terms.

Plan the exit

A training run ends, so the exit is easy to plan. An inference service does not, so the contract needs a way to scale down. Ask what happens if you need half the capacity in six months, or twice as much next quarter.

We handle both routes. GPU capacity is available for rent and hardware for purchase, through different partners, and the terms are agreed individually for each request: there is no fixed commission and no fixed minimum. Tell us the job, the GPU, the quantity, the location and the timing in the form below, and we come back with what is available and on what terms.

Can the same GPUs do both training and inference? Yes. The same families do both, and most clouds rent them for both. The choice is about fit: how much memory the model needs, how fast the GPUs talk to each other, and how many hours the job will run.

Questions and answers

What is the difference between AI training and inference?
Training is the one-off, very large job that produces a model, run for weeks on thousands of tightly connected GPUs. Inference is the service that uses the finished model, run for as long as the product exists, on as many or as few GPUs as demand needs.
Why is inference limited by memory rather than compute?
NVIDIA's engineers describe two phases. In prefill, the model processes the input tokens in parallel, which saturates the GPU. In decode, output tokens are generated one at a time, and NVIDIA calls this a memory-bound operation, because moving data from memory takes longer than the arithmetic.
How many GPU hours did it take to train Llama 3.1 405B?
Meta's model card for Llama 3.1 lists 30.84 million GPU hours on H100-80GB hardware for the 405B model, at 700 W per GPU, and 39.3 million GPU hours for the three sizes together.
Can the same GPUs do both training and inference?
Yes. The same families do both, and most clouds rent them for both. The choice is about fit: how much memory the model needs, how fast the GPUs talk to each other, and how many hours the job will run.

Sources: every document behind this guide is listed under Sources, AI Inference vs Training: Hardware, Power and Cost. Figures marked as Compute Census data come from the GPU price index and the campus census; how they are built: methodology.

Source GPUs for training or inference

Renting a block of GPUs for a training run or buying inference servers? Both are handled, through different partners, on terms agreed for each request. Tell us the model, quantity, location and timing.

What do you need

Handled by The Dream View Co., Ltd. (Phuket, Thailand), which arranges GPU rental, hardware purchase and hosting through different partners. Terms are agreed individually for each request. Hardware sales are subject to export-control rules. Data use: privacy notice.