How much GPU memory a model needs for its weights

The figure for a 70 billion parameter model

A model's weights need roughly its parameter count multiplied by the bytes each parameter is stored in. For a 70 billion parameter model that is about 140 GB in 16-bit, about 70 GB in 8-bit and about 35 GB in 4-bit. That is a floor, not a total. The key-value cache and the activations need memory on top, they grow with context length and batch size, and no single number covers them, so the real requirement is the weights plus headroom you measure on your own traffic.

Because this is arithmetic, it is the one part of choosing a card you can settle before spending anything. Throughput, latency and how a given serving engine behaves on a given card are things this site holds no data on, and the honest route to those is a test run on rented hardware. Memory is different. A model that does not fit will not load, and no amount of tuning changes that.

Bytes per parameter, worked through

In 16-bit precision each parameter takes 2 bytes. In 8-bit it takes 1 byte, and in 4-bit it takes half a byte. Multiply by the parameter count in billions and the answer comes out in gigabytes, near enough: a 7 billion parameter model is about 14 GB in 16-bit, 7 GB in 8-bit and 3.5 GB in 4-bit.

Scale it up and the pattern holds. A 13 billion parameter model is about 26 GB, 13 GB or 6.5 GB. A 34 billion parameter model is about 68 GB, 34 GB or 17 GB. A 405 billion parameter model is about 810 GB in 16-bit, which no single card on the list holds, and about 203 GB even at 4-bit, more than the 192 GB of a B200 or an MI300X.

Quantising to 8-bit or 4-bit halves or quarters the weights, and it changes the model. Whether the quality loss matters for your task is a question for your own evaluation set, and the answer differs by model, by method and by what you ask of it. The memory saving is certain; the cost of it is not something arithmetic can tell you.

What the weights do not include

Serving a model keeps a key-value cache for every token in every active request. Its size depends on the model's layer count and attention layout, the context length and how many requests run at once. A single short chat might need very little. Many long documents at once can need tens of gigabytes, and on a small model the cache can outgrow the weights.

Activations, the serving engine's own buffers and the CUDA context take more. Training and fine-tuning add gradients and optimiser state, which is a different and much larger bill. The practical habit is to load the model on a rented card, push it with your real context lengths and concurrency, and read the peak memory off the card rather than trusting a calculator.

One trap catches people who size by weights alone. Some serving engines reserve most of the card's free memory for the cache up front, so a monitoring tool shows the card nearly full the moment the model loads. That reading says nothing about whether the model fits; look at the engine's own report of how much cache it allocated and how many tokens that holds.

From a gigabyte figure to a shortlist of cards

Take the weights figure, add the headroom you measured, and look for the smallest card or set of cards that holds it. A 70 billion parameter model in 8-bit at about 70 GB sits on an 80 GB card such as the H100 with around 10 GB left for everything else, which is tight for long contexts. The same model in 16-bit at about 140 GB nearly fills the 141 GB of an H200 and leaves no room, so in practice it goes across 2 GPUs or onto a 192 GB part such as the B200 or the MI300X.

Every card the site prices is listed on the GPU page with its memory and its current rates, so the shortlist can be drawn from the cards you can actually rent rather than from a spec sheet. A card with no listed rate may still be available from someone who does not publish a price; it just cannot be compared here.

Precision also narrows the list. A model you intend to serve in 16-bit needs twice the memory of the same model in 8-bit, and a card that supports 8-bit floating point (Hopper and later) gives you a native way to run at 8-bit. Older cards can still hold quantised weights, through integer formats with their own trade-offs.

Pricing the memory you need

Once the shortlist exists, the rate decides the rest. On record, the cheapest hourly listing for the H100 SXM 80GB is $1.68, and for the H200 141GB, $2.09. For a model split across several cards, the hourly cost is the per-GPU rate times the number of cards, and the rental cost calculator runs that for any card and any number of hours.

Comparing a single large card against a pair of smaller ones is often the real decision. A pair costs a pair of rates and adds the work of splitting the model between them; one card with enough memory avoids the split. Which is cheaper per useful hour depends on your model and your traffic, so price both and test the one that wins on paper.

The arithmetic in this post is the same whichever provider you rent from. What changes between sellers is the rate, the terms and whether the card is free when you want it. Read those on the provider's page before committing to anything longer than a test.