LLM costs people forget until the invoice

The calls that never made it into the spreadsheet

The LLM costs people forget are the ones not tied to a visible user action: a system prompt resent on every call, chat history that grows each turn, retries after timeouts and bad output, evaluation runs, embedding jobs, and the extra model calls an agent makes between tool uses. Each is billed in tokens at the same rates as the answer the user sees, and together they can outweigh it.

None of these is exotic. They come from building a product the normal way, and they are invisible in a budget that counts one model call per click and multiplies by the input and output rates on the token price table.

The pattern behind all of them is the same. A budget counts the tokens a person can see, and the bill counts every token sent. The gap between those two is the subject of this post, and it grows as a product adds features.

Most of them also grow with success. More users means longer chat histories, more retries under load, a bigger evaluation suite and a larger index to embed. A forgotten cost that is small at launch grows with the user count.

Text that rides along on every request

The system prompt is the first. It starts short and collects rules, examples and formatting notes over months, and every one of those additions is multiplied by every request in production. Nobody notices because no single request looks expensive.

Chat history is the second. A stateless API needs the conversation resent each turn, so the tenth message carries the nine before it. Tool definitions are the third: an agent with many tools sends every tool's description and schema as input, on every step, whether it uses them or not.

Where a seller publishes a cached input rate, a fixed prefix like a system prompt or tool list can be billed lower. Listings with such a rate on record: 507. A listing without one bills the repeat at the full input rate.

Calls nobody sees

Retries are billed. A request that times out after the model has started generating may still be charged, depending on the seller. A response that fails JSON validation and is sent again costs twice. A user pressing regenerate costs again. Backoff spaces retries out, and each one is still billed.

Agents multiply calls. One user request to an agent can become a plan, several tool calls each followed by a model call to read the result, and a final answer. Each step resends the growing history.

Guardrail and routing calls count too. A classifier that checks every input for safety, or a small model that decides which large model to call, is a model call with its own tokens.

Work that happens off the request path

Evaluation is easy to forget. Every prompt change gets tested against a suite of examples, often on several models, and often with a second model grading the output. A suite of a few thousand cases run on each change adds up across a busy month. The evaluation category lists tools built for this work.

Embeddings are priced per token too. Building a retrieval index means embedding every document, and changing the embedding model or the chunking means doing it all again. Development traffic, staging environments and the engineer testing prompts by hand all draw on the same account unless they are separated.

Fine-tuning has its own line where a seller offers it, often priced per training token, and calling the tuned model afterwards can carry a higher rate. A tuned model that ends up handling all traffic moves every request onto that higher rate, which belongs in the estimate before the tuning starts.

Finding them before the bill does

Tag every call with the feature that made it, and log the input and output tokens the API returns. Group the totals by tag weekly. The forgotten costs show up as tags with large totals and no user-facing feature attached.

Then estimate in two lines. The first is the visible calls: requests times tokens times rates. The second is everything else, measured from those logs as a share of the first. Put both totals into the LLM cost calculator, so the monthly figure includes what actually gets billed.

Set a budget alert on the provider account as well. It will not explain a spike, but it will tell you on the day rather than at month end.

Review the tags monthly with whoever owns the budget. A line that grew without a product change behind it is the one to explain first, because it is often a prompt, a retry loop or a test job that nobody meant to keep running.

A short audit to run this week

Open the provider dashboard and the usage logs side by side. Find the share of tokens that came from requests with no user attached: scheduled jobs, evaluation runs, staging. Then find the longest system prompt in production and multiply its token count by last month's request count.

Those two figures can explain much of the gap between a budget and an invoice. Neither needs new tooling, only the token counts the API already returns and a few minutes with a spreadsheet.

Fix the largest one first. Shortening a system prompt that rides on every call, or moving evaluation runs to a batch tier where the seller offers one, changes the bill from the following day without touching what users see.