24 GB, 48 GB or 80 GB: which memory tier

Choose by what has to fit

Rent the smallest memory tier that holds your model's weights plus the headroom you measured for its cache and activations. In rough terms, 24 GB suits models up to about 7 or 8 billion parameters in 16-bit, 48 GB suits up to about 13 billion in 16-bit or 34 billion in 8-bit, and 80 GB is where a 70 billion parameter model fits in 8-bit. The headroom depends on context length and concurrency, so the boundaries move with your traffic.

Memory tiers are a useful way to shop because the cards inside one tier hold the same models. Inside a tier the choice becomes price, generation and terms. Across tiers it is a question of whether your model fits at all.

This site holds no speed figures for any card, so it cannot rank cards within a tier by how fast they serve a model. It can show what each tier holds by arithmetic and what each card is listed at, and the test on your workload does the rest.

The 24 GB tier

Cards here include the L4, the A10, the RTX 4090 and the RTX 3090, with the RTX 5090 a step above at 32 GB. Cheapest hourly listings on record: L4, $0.33; A10, $0.24; RTX 4090, $0.12.

At 2 bytes a parameter, 24 GB holds about 12 billion parameters of weights and nothing else, so in practice it is a 7 or 8 billion parameter tier in 16-bit. Quantised to 4-bit, a 32 billion parameter model at about 16 GB fits with a little room. Embeddings, small chat models, speech, image generation and evaluation runs often fit in this tier.

The L4 and the A10 are data-centre cards; the RTX cards are consumer parts, more often found on marketplaces with terms set by each host. For a short test the difference may not matter. For a service, read the terms before the rate.

The 48 GB tier

The L40S, the L40, the A40 and the RTX 6000 Ada carry 48 GB. Cheapest hourly listings on record: L40S, $0.38; A40, $0.52. These are PCIe cards with graphics memory, sold for inference, rendering and mid-size fine-tuning.

This tier is where a 13 billion parameter model in 16-bit sits comfortably, with around 22 GB beyond its weights, and where a 34 billion parameter model fits in 8-bit. A 70 billion parameter model fits only at 4-bit, which leaves little room for a cache. The L40S post works through three serving cases in this tier.

The 80 GB tier, and above it

The A100 80GB and the H100 in SXM and PCIe form hold 80 GB. Cheapest hourly listings on record: H100 SXM 80GB, $1.68; A100 SXM 80GB, $0.41. A 70 billion parameter model in 8-bit at about 70 GB fits here with a tight margin, and a 32 billion parameter model in 16-bit, about 64 GB, fits with about 16 GB to spare.

Beyond 80 GB are the RTX PRO 6000 at 96 GB, the GH200 at 96 GB, the H200 at 141 GB, and the B200 and MI300X at 192 GB. They are for models that need more than one 80 GB card in the precision you want, or for caches that outgrow it.

A pair of cards in a lower tier can sometimes replace one card in a higher tier. That works when the serving engine supports splitting the model, and costs a link between the cards that matters more on PCIe than on SXM.

What moving between tiers costs

Stepping down a tier is rarely just a cheaper rate. If the model only fits the smaller card at 8-bit or 4-bit, you are now serving a quantised model, which needs its own quality check, its own conversion step and sometimes a different serving engine. Budget an afternoon for that before counting the saving.

Stepping up is simpler in software and dearer per hour. A model that fitted at 4-bit on 48 GB can run at 8-bit on 80 GB, and the cache gets room to grow. Whether that improves your output enough to justify the rate is a judgement made on your own evaluation set.

Splitting a model across several smaller cards sits between the two. It keeps the precision you want on cheaper cards, at the price of a parallel setup and traffic between them. On PCIe cards that traffic takes the slower path, so test it under load before treating a pair of 48 GB cards as one 96 GB card.

Finding the tier for your traffic

Start from the weights figure: parameters in billions times bytes per parameter. The post on memory for weights has the arithmetic. Then rent a card in the tier you think fits and load the model with your longest real prompts and your busiest hour of concurrent users. The peak memory reading tells you whether the tier is right.

A peak well under the card's memory is a reason to try the tier below. If it is within a few gigabytes of the limit, step up, because traffic is never as tidy as the test. Every tier is listed on the GPU rate table with the date each rate was read.

Repeat the test when the traffic changes. A tier chosen for a pilot with a handful of users can be wrong a quarter later, in either direction, and the reading takes an hour.