Cached input tokens: when they lower the bill

Caching lowers the bill when the start of every prompt repeats

Cached input tokens lower an LLM bill when many requests begin with the same long prefix and the seller publishes a cheaper rate for reading that prefix again. The provider keeps the processed prefix for a while, and a later request that starts identically is billed at the cached rate for that part. Where no cached rate is published, there is no discount: every input token pays the standard rate.

Input-price listings on record: 750. Listings among them with a cached input rate: 507. A listing without one gets no caching discount, so caching cannot be assumed just because a model is popular or a seller is large. Check the token price table for the exact model and seller you plan to call.

Caching is a property of the seller's serving system, not of the model. The same model can carry a cached rate at one seller and none at another, which can matter more to a long-prompt workload than the gap between their standard input rates.

Which workloads repeat a prefix

The obvious case is a long fixed system prompt: policies, tone rules, worked examples, a tool schema. Every call starts with the same block, and only the user's message changes at the end.

Document question answering is the second. A user uploads a manual and asks ten questions about it. With the manual at the start of each request and the question after it, the manual is a shared prefix across all ten.

Agents running a long task fit too, since each step resends the history so far and appends one new action. The earlier steps repeat; only the tail is new. A batch of unrelated one-off prompts, by contrast, shares nothing and gains nothing from a cache.

Retrieval systems sit in between. The same few documents, retrieved for many users and placed in a fixed order after the instructions, can form part of a shared prefix. Where each request retrieves different passages, only the instructions above them repeat.

Order is what makes a prefix cacheable

A cache matches from the first token forward and stops at the first difference. Put a timestamp, a user name or a request ID at the top of the prompt and nothing after it can match, however much of the rest is identical.

The fix is ordering: fixed instructions first, then slow-changing context such as a document, then the part that changes on every call. That layout costs nothing to adopt and decides whether caching applies at all.

Cached prefixes also expire. Sellers that offer caching can keep entries for a limited time, so traffic that arrives in steady streams hits the cache more often than traffic that arrives in rare bursts. The expiry rules are the seller's and are not something we record.

The arithmetic with a cached share

Split monthly input tokens into two parts. The cached part is total input times the share that repeats, billed at the cached rate per million. The fresh part is the remainder, billed at the standard input rate. Output is unchanged. Add the three.

Take a coding assistant whose every request opens with a long repository summary and ends with a short question. The summary is most of the input and repeats on every call within a session, so the cached share is high and the cached rate carries most of the input bill. Move the request ID above the summary and the share drops to nothing.

The LLM cost calculator has a slider for the cached share of input. It applies that share only to sellers that publish a cached rate and bills the others at the full input rate, which the table states under the results.

What the cached figure leaves out

Some sellers charge to write a prefix into the cache, separately from the rate for reading it back. A prefix used once then pays the write cost and never earns it back. Where a seller does this, a low cached share can cost more than no caching.

A cached share is also an estimate. The honest way to get it is to log the cached token count the API returns with each response for a week and divide by total input. Until then, try the calculator at a low share and a high one and treat the gap as the range you are working in.

Caching does nothing for output. A workload whose bill is mostly the answer side gains little from any cached rate, however high the share. Look at which half of the bill is larger before spending time reordering prompts.

Checking the cache did what you planned

After switching on caching, compare a week of usage before and after. The usage block on each response reports how many input tokens were read from cache. Divide that by total input tokens for the week to get the real cached share, and put it back into the estimate in place of the guess.

A share far below what you expected usually means the prefix is not identical across calls. Common culprits are a date inserted into the instructions, tools listed in a different order, or a retrieved document placed above the fixed text. Each one breaks the match from that point on.

A share close to what you planned, but a bill that barely moved, usually means input was never the large half. On an output-heavy workload a cheaper input rate changes little, and the effort is better spent on shortening what the model writes.