Quantization: what you trade for the smaller model

Fewer bits per weight, less memory per model

Quantization stores a model's weights at lower precision, typically 8-bit or 4-bit in place of 16-bit, so the model needs less GPU memory. A 70 billion parameter model takes about 140 GB in 16-bit, about 70 GB in 8-bit and about 35 GB in 4-bit, before cache. What you trade is precision: every weight is rounded, and the model's answers can change. How much they change for your task is something only your own evaluation can tell you.

That is the whole bargain. The memory saving is arithmetic and can be worked out on paper. The quality cost is empirical and cannot.

So the order of work matters. Do the memory arithmetic first to find which precisions would let the model fit the hardware you can afford, then test only those. Testing every format on the hub is a week; testing the two that change your bill is an afternoon.

The arithmetic that makes it attractive

Weights need about the parameter count times the bytes per parameter. In 16-bit that is 2 bytes, in 8-bit 1 byte, in 4-bit half a byte. A 7 billion parameter model drops from about 14 GB to about 7 GB to about 3.5 GB across those steps.

Those drops move a model across hardware boundaries. In 4-bit, a 70 billion parameter model's weights fit on one H100 80GB with room for some cache, where in 16-bit they need a second card. A 13 billion parameter model in 4-bit (about 6.5 GB of weights) fits on an RTX 4090 24GB. The lowest listed hourly rates are $0.12 for the RTX 4090 and $1.68 for the H100 SXM 80GB. The rates for every card are on the GPU rates page.

The arithmetic leaves out the key-value cache or activations, which take memory on top, grow with context length and batch size, and cannot be given as one figure. A model that barely fits by weight will not fit once real requests arrive. Leave room for the cache first and pick the precision second, not the other way round.

Where the format names come from

Model hubs list the same model many times over, with suffixes that name a bit width and a quantization method. The suffix tells you the storage format, and usually which engine or library can load it. It does not tell you how the rounding was calibrated, which sample of text was used to choose the scales, or whether anyone tested the result on a task like yours.

Two files with the same bit width can behave differently because they were made by different methods. Some methods look at sample data and protect the weights that matter most to the output; others round every weight the same way. That is a reason to treat each file as its own model to evaluate, not as a smaller copy of the original.

Check who published a quantized file too. A version from the model maker comes with their own notes; a community upload may be excellent or may be untested. Neither is a verdict, and each gets the same run through your test set.

What rounding does to a model

A 16-bit weight can take many more distinct values than a 4-bit one. Quantizing maps each weight to the nearest value the smaller format can hold, usually with a scale factor per group of weights so the rounding error stays small. The model is the same shape; its numbers are coarser.

Some tasks barely notice. Others do: long chains of reasoning, arithmetic, code, and languages that were thin in the training data are where people tend to look first for damage. We hold no quality measurements, so we cannot say where your model will slip. We can say that a single headline score from someone else's benchmark is weak evidence about your prompts.

Failures can be quiet: a plausible answer that is wrong a little more often, or an output format that occasionally breaks a parser. Problems like that surface in production logs weeks later, so the test set should include the structured outputs your code depends on.

Weights, activations and the KV cache are separate choices

Weight-only quantization shrinks the stored model and leaves the arithmetic at higher precision. Quantizing activations too changes the arithmetic itself, which depends on the card: Hopper and later support 8-bit floating point arithmetic, and older cards do not support it natively.

The key-value cache can also be stored at lower precision. That shrinks the memory each request holds, which raises how many requests fit at once. It is a different trade from weight quantization and deserves its own test, since errors in cached context can compound over a long reply.

Inference engines differ in which formats they support. The inference engine category lists the products, and each product page links the vendor's own documentation for formats.

How to decide without a benchmark from us

Build a test set from your own traffic: a few hundred real prompts with answers you would accept. Run the 16-bit model and each quantized version against it, and grade the outputs the same way. A smaller model that passes your bar gives you the memory saving at no cost you can find. One that fails has told you so cheaply.

Then price the hardware each version needs with the GPU cost calculator. A quantized model that lets one card do the work of a pair halves the rental line on the bill, and that is the number to set against whatever quality gap your test found.