Explaining cached input pricing so buyers can use it

A discount nobody can plan for

Cached input pricing is a lower rate for input tokens the provider has seen recently, and a buyer can only count on it if the page says three things beside the rate: what makes input count as cached, how long a cached prefix stays cached, and whether the buyer must mark it or it happens on its own. A rate without those answers is a discount that may or may not arrive.

Of the 750 input-price listings we hold, 507 also state a cached input rate. That count says how many pages publish the number. It says nothing about how many explain it.

What counts as cached

Caching usually applies to a repeated prefix: the same system prompt, the same long document or the same tool definitions sent at the start of many requests. The page needs to say whether the match has to be exact from the first token, whether there is a minimum length before anything is cached, and whether a change halfway down invalidates the rest.

These details decide whether an application can be arranged to benefit. A team will move its stable instructions to the top and its user text to the end if the page tells them that is what earns the rate. Without that, the team guesses, measures the bill and guesses again.

Say also whether the cache is shared across an organisation or kept per key, and whether it crosses models. A team running several services on one account will plan differently depending on the answer.

An example request helps most here, short enough to read in a minute, with the cached part marked. One example does more than a page of rules, because it shows the shape the rules describe.

How long the cache lives

A cache that expires after a short idle period helps a busy chatbot and does nothing for a nightly job. A buyer cannot tell which they are buying unless the page gives the lifetime, says whether each hit renews it, and says whether a longer lifetime is available at another price.

Writing to the cache is sometimes priced too, above the normal input rate. That changes the arithmetic: a prefix sent once and never reused costs more than it would have uncached. State the write price where there is one, on the same row, so the buyer can see the break-even without reading the docs.

Lifetime also decides who benefits. A large customer with steady traffic keeps the cache warm all day and gets the rate almost always. A small customer with bursts may rarely get it. If the rate is part of your pitch to small teams, the lifetime is the honest test of that pitch.

Automatic or requested

Some caching happens without the buyer doing anything, and some needs a marker in the request. The difference matters for a buyer estimating cost before they have written any code. An automatic cache is a discount they will get if their traffic repeats. A requested one is a discount they will get after an engineer reads the docs and changes the client.

Say which it is in one sentence next to the rate. Link to the exact section of the documentation that shows the request, rather than the documentation home page.

If both exist, say which applies by default and what the requested kind adds, such as a longer lifetime. With both visible, a team can pick the cheaper path for each feature.

A worked case, without the numbers

Take a support assistant. Every request carries the same long block of product documentation and the same instructions, followed by a short customer question. Almost all of the input is the repeated block, so the cached rate applies to most of each request if the cache is still warm when the next question arrives.

Now take the same assistant on a quiet weekend, with an hour between questions. If the cache lifetime is shorter than that gap, every request pays the write price again and the cached rate never applies. The rate on the page is the same in both cases. The bill is not, and only the lifetime explains the difference.

A page that gives the lifetime lets the buyer run this case for their own traffic before signing. A page that gives only the rate lets them run it after the first invoice, which is when they decide whether to trust the rest of the page.

Where cached rates go wrong in a comparison

The cost calculator works from published rates only. A seller that caches but does not publish a cached rate is therefore priced from its page as if it did not cache, which is a fair reading of the page and an expensive one for the seller.

The other failure is a cached rate stated for some models and not others without saying why. A reader has no way to tell the gap from no caching. If a model is cached at the normal rate, or not cached at all, say so on its row; on our token price table a blank reads as no answer on record.

Cached rates that change by model family, or apply only above a certain prefix length, belong on the row of each model they apply to. A buyer filtering for a cached rate reads the rows, never a paragraph explaining the exceptions.