Observability and evals for LLM apps

Tracing records what each model call did; evaluation scores whether it was right. An LLM app needs both, and they answer different questions at different times.

INFRATOAPP · PUBLISHED 21 OCTOBER 2026 · 5 MIN READ · 3 SOURCES · PRICES LAST CHECKED 18H AGO

Tracing tells you what happened; evals tell you if it was right

LLM observability has two halves. Tracing records each request as it happens: the prompt, the retrieved context, every model and tool call, token counts, latency and cost. Evaluation scores outputs against a standard, either on a fixed test set before a release or on samples of live traffic after it. Tracing answers what the system did. Evaluation answers whether that was any good.

The observability layer holds products for both (live products on record: 68), and many do some of each. Knowing which half a product is strong in matters more than its feature list, because the two are used by different people at different moments.

An engineer opens a trace when a customer complains about one answer. A product lead opens an evaluation report when deciding whether the new prompt ships. Where one tool serves both, check that each view is good enough for the person who will use it, not only that the feature exists.

What a useful trace contains

A trace should let someone who was not there replay a bad answer. That means the full prompt as sent, with the template version and the variables filled in, the passages retrieved and their scores, each tool call with arguments and result, and the final output. Token counts per step tell you where the money went.

For agents the trace is a tree, not a line. One user request can spawn a planning call, several tool calls, a retry and a final summary. A flat log of model calls hides which one went wrong, and you will spend an afternoon reconstructing the order by timestamp.

Decide early what you must not store. Prompts often contain customer data, and a tracing tool that keeps everything forever is a new place that data lives. Redaction before the trace leaves your service is easier than deletion after.

Evaluation before release: a test set you trust

Offline evaluation runs a fixed set of inputs through the system and scores the outputs. Start with real questions from your logs, write down what a good answer contains, and keep the set small enough that someone reads every failure. A set nobody reads becomes a number that drifts without anyone asking why.

Scoring comes in a few forms. Exact checks work for structured output: valid JSON, the right label, a figure within range. Comparisons against a reference answer work for factual questions. A second model acting as judge can score looser qualities such as relevance or faithfulness to sources, and its scores need checking against a human sample before you trust them.

Evaluation tools run these sets, store the scores per prompt and model version, and show what changed. The value is in the comparison: this prompt against the last one, this model against the cheaper one.

Evaluation after release: sampling live traffic

Online evaluation scores a sample of real traffic, because users ask things your test set never imagined. The same judges can run on sampled traces, and user signals such as a thumbs down, a retry or an abandoned session point at answers worth reading.

Feed what you find back into the offline set. Every bad live answer that a person confirms is a test case that stops the same failure returning. Over months, that loop is how the test set comes to reflect real use.

Watch for drift you did not cause. A hosted model can change behind the same name, and a provider can retire the version you tested. Scores on sampled traffic that fall with no deploy on your side are often the first sign.

Cost and latency belong in the same view

Quality is one axis. Every trace also carries tokens and time, and a change that improves answers while doubling the input tokens is a trade someone should make on purpose. Put cost per request and latency beside the evaluation scores, broken down by prompt version and model, so the trade is visible at the moment of the decision.

Slow steps hide in agents and retrieval. A trace tree with timings on each node shows whether the delay is the model, a reranker, a tool that waits on a third-party API, or a retry loop nobody meant to write. Averages hide all of it; look at the slow tail, where the complaints come from.

Group costs by feature and by customer where you can. A jump in spend can come from one heavy customer or one chatty feature, and a trace store that can be cut that way answers the question in minutes rather than a day of spreadsheet work.

Questions to ask before choosing a tool

Ask how it captures traces: an SDK in your code, a proxy in front of the model, or an integration with the framework you already use. A proxy is quick to add and sees only model calls. An SDK sees your retrieval and tool steps too, at the price of code changes.

Ask whether it accepts traces in an open format. When your framework, gateway and tracing tool agree on one, you can change any of them without losing history, and you are less tied to the first vendor you picked in a hurry.

Ask where the data is stored and for how long, whether you can self-host, and whether you can export traces and scores. Then ask how it prices: per trace, per seat or per stored event. Trace volume grows with traffic, and a per-trace price that was trivial at launch can be a real line on the bill later. Check the layer page for the products on record and confirm pricing with the vendor.

What we could not answer

⚑We hold no data on the accuracy of model-based judges, so nothing here says how far their scores can be trusted for a given task.
⚑Pricing for observability tools is not tracked on this site, so the cost advice is about structure only.

What this rests on

infratoapp, “Observability and evals for LLM apps”, read 27 October 2026. products in the observability layer: 68. https://www.infratoapp.com/blog/observability-and-evals-for-llm-apps

Questions people ask about this

What is the difference between LLM observability and evaluation?

Observability, or tracing, records what each request did: prompts, retrieved context, model and tool calls, tokens, latency and cost. Evaluation scores whether the outputs were right, against a test set before release or on sampled live traffic after. Tracing helps debug a single bad answer; evaluation shows whether a change made answers better or worse overall.

Can an LLM judge the output of another LLM?

It can score loose qualities such as relevance or whether an answer is supported by its sources, and it scales where human review cannot. Its scores need checking against a sample graded by people before you rely on them, and exact checks should be used wherever the output has a structure that can be tested directly.

When should I add observability to an LLM app?

Before the first user outside the team. Without traces there is no way to explain a bad answer, see what a day cost, or tell whether a prompt change helped. A simple log of prompts, outputs, model, tokens and latency is enough to begin, and a dedicated tool can replace it later.

These render as FAQPage structured data · answers are filled from the same live figures