Latency or throughput: which to tune for

Who is waiting decides it

When a person is watching a cursor blink, tune for latency: the time to the first token and the gap between tokens. If the output lands in a database that someone reads tomorrow, tune for throughput: tokens produced per GPU-hour, which is what sets cost. The two pull against each other through batching. Bigger batches use the GPU better and lower cost per token, and make each request wait longer.

Most systems have both kinds of traffic, and a common mistake is serving them from one pool tuned for neither.

The pager tends to find this out first. A nightly job lands on the same replicas as daytime chat in another timezone, batch sizes swell, and someone in a support channel reports that the assistant has gone slow. Nothing is broken; two goals are sharing one setting.

Why batching moves both numbers

Generating a token means reading the model's weights from memory and doing arithmetic with them. For one request at a time, the GPU spends much of its effort moving weights and little on arithmetic. Group many requests and the same weight read serves all of them, so more tokens come out per second of GPU time.

The cost is waiting. A request may sit until a batch forms, and once in a batch it shares each step with others. Continuous batching, where requests join and leave between steps, narrows that wait without removing it. Every engine in the inference engine category makes some choice here, and the knobs they expose are where you set the trade.

Batches also hold memory. Each request in flight keeps its own key-value cache, so a larger batch needs more free memory beside the weights. The batch size you can afford is bounded by the card before it is bounded by your patience.

Prefill and decode pull in different directions

A request has two phases. Prefill reads the whole prompt in one pass, and its cost grows with prompt length. Decode produces the reply a token at a time, and its cost grows with output length. A long prompt delays the first token; a long reply stretches the time after it.

They load the card differently. Prefill does a lot of arithmetic on many tokens at once and keeps the GPU busy even for one request. Decode does a little arithmetic per step and leans on memory reads, which is where batching pays. When a big prefill lands in the middle of other requests' decoding, those users see a stall in their stream.

Approaches to that collision include splitting long prefills into chunks and running the two phases on separate cards, and engine documentation says which an engine offers. Which of those helps depends on your mix of prompt and reply lengths. Know that mix before reading anyone's tuning advice, including this post.

Interactive traffic: what to protect

Chat, code completion and voice agents are judged by how quickly the first words appear and how steadily the rest follows. Streaming exists for this traffic: a reply that starts quickly and arrives in pieces gives the user something to read while the rest is generated.

For this traffic, cap the batch size, keep spare capacity so a burst does not queue, and watch the slowest requests, the tail percentiles, before the average. An average can look fine while a few users wait long enough to leave. Spare capacity costs money every hour, and the bill should show it as the cost of responsiveness.

Shorter outputs help too. A reply capped at the length the interface can show is cheaper to generate and finishes sooner, and a prompt that asks for brevity is the only latency fix that costs nothing.

Offline traffic: what to squeeze

Summarising a document archive, tagging a catalogue, generating embeddings or running an evaluation suite has no human in the loop. Here the right setting is the largest batch the card can hold, run until the queue is empty, and then switch the card off.

Many API providers sell exactly this as a batch tier: submit a file, accept a delay, pay less per token. The LLM prices page shows which listings carry a batch price (the count on record is 121, against 750 input-price listings). Offline work that runs through an API should be checked against a batch price before anything is built.

On rented GPUs, offline work suits interruptible capacity, since a reclaimed card costs a restart, not a customer. The rates for each card are on the GPU rates page.

Write offline jobs so they checkpoint progress, then a reclaimed card loses minutes of work rather than the whole run, and the cheaper capacity stays cheap in practice.

Splitting the two in practice

The cleanest set-up is two pools. One serves interactive requests with small batches and headroom. The other drains a queue of offline jobs with large batches and no headroom. Each is tuned for one goal, and neither drags the other.

With budget for only one pool, give interactive traffic priority and let offline jobs fill the gaps. A scheduler that admits batch work only when the interactive queue is short keeps offline jobs moving without making the interactive queue wait behind them.

The figures to collect are your own: time to first token, tokens per second per request, and tokens per GPU-hour, measured on your model and your prompts. We publish none of them, because a number measured on someone else's traffic tells you little about yours.