A GPU for fine-tuning or for inference: different needs

Training state against a serving cache

Fine-tuning and inference fill GPU memory with different things. Full fine-tuning keeps the weights, a gradient for every weight, and the optimiser's running state, which together can be several times the size of the weights alone. Inference keeps the weights and a key-value cache that grows with context and concurrency. So a card that serves a model easily can be far too small to fine-tune it, and adapter methods that freeze the base model close most of that gap.

Neither need can be reduced to one figure. The cache depends on traffic and the training state depends on the method and its settings. What follows is the arithmetic for each, stated with its assumptions, and the questions it leaves for a test.

This site holds no training benchmarks and no serving benchmarks. It can do the memory sums and show listed rates, and the rest depends on your model, your data and the method you pick, which is why the method has to come first.

Full fine-tuning, by the bytes

Take a 7 billion parameter model trained in mixed precision with an Adam-style optimiser, a common setup. The 16-bit weights are about 14 GB. The 16-bit gradients are another 14 GB. Many setups keep a 32-bit master copy of the weights, about 28 GB, and the optimiser keeps two 32-bit values per weight, about 56 GB. That is about 112 GB before activations, the common estimate of 16 bytes per parameter, for a model whose weights alone are 14 GB. Treat it as an estimate for this setup, not a specification: a different optimiser, precision or framework changes it.

Activations come on top and grow with sequence length and batch size. Techniques such as recomputing activations, sharding optimiser state across GPUs or using an 8-bit optimiser cut the total, each with its own cost. The sum gives the order of magnitude: in this setup a full fine-tune of a 7 billion parameter model needs more than one 80 GB card, or one of those techniques to bring it under 80 GB, and is far out of reach of a 24 GB card.

Scale the same sum to a 70 billion parameter model and the training state alone runs past a terabyte. In this setup that is a job spread over many GPUs, and the size of that bill is the reason adapter methods exist.

Adapters change the sum

Adapter methods such as LoRA freeze the base model and train a small set of extra weights. Gradients and optimiser state are needed only for those, so the memory beyond the frozen weights is small. A 7 billion parameter base in 16-bit is still about 14 GB, plus the adapter's share and activations.

Quantising the frozen base to 4-bit, as QLoRA does, cuts its weights to about 3.5 GB for 7 billion parameters and about 35 GB for 70 billion. So fine-tuning a 70 billion parameter model with adapters can fit on one 48 GB or 80 GB card, where a full fine-tune of the same model would need many.

Whether an adapter reaches the quality of a full fine-tune for your task is something only your own evaluation shows.

What inference asks of the same card

Serving drops gradients and optimiser state entirely. The weights stay, in whatever precision you serve them, and the cache grows with every active request. A 7 billion parameter model in 16-bit serves from a 24 GB card with room for moderate traffic; the same model needed about 112 GB to fully fine-tune.

So a common pattern is to fine-tune on a large card for a few hours and serve on a smaller one for months. The rates make that concrete: the H100 SXM 80GB is listed from $1.68 an hour and the L40S 48GB from $0.38, with every card on the GPU rate table. The rental cost calculator prices a short training run and a month of serving separately.

Training hours against serving hours

A fine-tuning run is short and heavy. It might take hours or a few days on a large card, and then it is over. Serving runs every hour of every month at whatever load arrives. Those two shapes of spending reward different choices.

For training, a higher hourly rate on a card with enough memory is often fine, because the hours are few and running out of memory halfway through costs the whole run. Interruptible capacity can suit it too, if the job saves checkpoints often enough that losing the machine costs minutes, not days.

For serving, the hours are many, so the rate dominates. It pays to find the smallest card that holds the weights and the cache for your traffic, and to quantise if quality allows. A model fine-tuned on an 80 GB card can often be served from a 24 GB or 48 GB one.

Choosing for the job you actually run

Write down the method before choosing a card: full fine-tune or adapter, which precision for the base, what sequence length. Work out the memory with the sums above, then rent a card one size up for a short trial run and read the peak. For serving, do the same with your real prompts and concurrency.

While the rest of the system is still undecided, the stack builder lays out the layers a fine-tuned model sits in, from the compute it trains on to the inference engine that serves it. The post on memory for weights covers the starting arithmetic in more detail.

Keep the fine-tuning and serving decisions separate in your budget too. One is a project cost with an end date, and the other is a running cost that grows with your users, so a saving on each is worth a different amount of effort.