The KV cache: why context costs memory

Memory that grows with every token

When a transformer model processes a request, every token, in the prompt and in the reply so far, leaves a key and a value vector in each layer of the model. The model reads them back at every later step, so they stay in GPU memory until the reply finishes. That store is the KV cache. It grows in a straight line with context length and again with the number of requests in flight, which is why a card that holds the weights comfortably can still run out of memory under load.

It is also why nobody can give you one figure for it. The size depends on the model, the context, the batch and the precision the cache is kept in.

Why the model keeps the keys and values

Attention lets each new token look back at every token before it. Doing that from scratch would mean recomputing the whole history at every step, which gets slower the longer the reply. Caching the keys and values of earlier tokens means each step only computes the newest token and reads the rest from memory.

That trade, memory for compute, is the right one almost always. It does mean memory per request is proportional to how many tokens that request has seen. A request with a long pasted document holds far more cache than a short question, even if both get short answers.

What sets the size, in words

For one request the cache is roughly: the number of tokens, times the number of layers, times two (a key and a value), times the width of each key and value, times the bytes per number. Models that share key and value heads across query heads keep a narrower cache than the parameter count would suggest, so models with the same parameter count can need very different amounts of cache.

For the whole card, multiply by the number of concurrent requests. Memory for the weights is fixed; memory for the cache is the part that moves with traffic. A 34 billion parameter model in 16-bit needs about 68 GB of weights, which leaves only a small margin on an 80 GB card and much more on an H200 141GB. The lowest listed hourly rate for the H200 is $2.09, on the H200 page. That margin is what the cache lives in.

Storing the cache in 8-bit instead of 16-bit halves its size. That is a separate choice from quantizing the weights, with its own effect on output quality to test.

Why a bigger card is sometimes the cheaper choice

Because the cache is what limits how many requests a card holds at once, memory beyond the weights converts directly into concurrency. A card with a thin margin over the weights serves few requests at a time however fast it computes. A card with a wide margin serves many more.

That makes cost per request, not cost per hour, the figure to compare. A dearer card that holds several times the concurrent requests can cost less per reply than a cheaper card that fits the weights and little else. The rates are on the GPU rates page; the concurrency each card supports for your model is something only a load test will tell you.

The same logic argues against splitting a model across more cards than its weights need just to gain cache room, unless the link between those cards is fast. Spreading work over a slow link can cost more in waiting than the extra memory buys.

The test needs no benchmark from anyone else. Run your own prompts at rising concurrency on each candidate card, note where requests start to queue, and divide the hourly rate by the replies served in that hour. The card with the lowest figure is the one to rent.

What happens when it runs out

An engine that cannot fit another request's cache has a few options, and none of them is free. It can queue the request, which adds wait. It can evict a request in flight and recompute it later, which wastes work. It can move cache to CPU memory and back, which is slow. Or it can refuse, and your client retries.

This is the usual cause of a serving system that looks fine in testing and degrades in production: tests used short prompts, production users paste whole files. Measure with your real prompt length distribution, the long tail included, before sizing the card.

Watch the engine's own metrics for this. Most report cache use, queued requests and evictions, and a rising eviction count under steady traffic is the earliest sign that the card is too small for the context your users bring.

Context limits deserve a product decision, not just an engineering one. A model that advertises a very long context window will accept a request that fills it, and that one request can hold as much cache as dozens of short ones. Decide what your feature needs and enforce it before the request reaches the engine.

Ways to spend less of it

Cap the context length you accept to what the product truly needs. Paged allocation, where an engine offers it, stops long requests from stranding blocks of memory. Sharing the cache for identical prompt prefixes, such as a long system prompt every request carries, stores that prefix once.

Some hosted APIs pass part of that saving on as a cached input price, shown on the LLM prices page (listings with one on record: 507). Requests that repeat a long prefix should be planned around the cached rate. See also quantization for what lower precision trades.