Observability / Evals

What a system did in production: spans, latency, token counts, errors and the trace of a single request. Whether the answer was any good is evaluation and benchmarking — eight products do both and are filed in both.

Ordered by attributes answered — the column we hold for most of them. Blank where we hold none.

langfuse10 ATTRIBUTES ANSWEREDbifrost8 ATTRIBUTES ANSWEREDLangSmith8 ATTRIBUTES ANSWEREDopenlit8 ATTRIBUTES ANSWEREDopik8 ATTRIBUTES ANSWEREDclawmetry7 ATTRIBUTES ANSWEREDlangwatch7 ATTRIBUTES ANSWEREDTruLens7 ATTRIBUTES ANSWEREDArize Phoenix6 ATTRIBUTES ANSWEREDFiddler AI6 ATTRIBUTES ANSWEREDHoneyHive6 ATTRIBUTES ANSWEREDHelicone5 ATTRIBUTES ANSWERED

Prompts as versioned artefacts: storage, diffing, templating, rollout and rollback. Measuring whether a change helped is evaluation.

Ordered by attributes answered — the column we hold for most of them. Blank where we hold none.

langfuse10 ATTRIBUTES ANSWEREDopenlit8 ATTRIBUTES ANSWEREDLloop-engineering2 ATTRIBUTES ANSWEREDSsystem_prompts_leaks2 ATTRIBUTES ANSWEREDacontext1 ATTRIBUTE ANSWEREDAagents1 ATTRIBUTE ANSWEREDCcl4r1t4s1 ATTRIBUTE ANSWEREDCclaude-skills1 ATTRIBUTE ANSWEREDlanggpt1 ATTRIBUTE ANSWEREDprompt-optimizer1 ATTRIBUTE ANSWEREDRig1 ATTRIBUTE ANSWEREDsuperpowers-zh1 ATTRIBUTE ANSWERED

Attributing spend to a team, a feature or a request, and showing where it went. Published rates by vendor are the price index; this is your own bill.

Ordered by attributes answered — the column we hold for most of them. Blank where we hold none.

langfuse10 ATTRIBUTES ANSWEREDLangSmith8 ATTRIBUTES ANSWEREDopenlit8 ATTRIBUTES ANSWEREDclawmetry7 ATTRIBUTES ANSWEREDTruLens7 ATTRIBUTES ANSWEREDFiddler AI6 ATTRIBUTES ANSWEREDlitellm3 ATTRIBUTES ANSWEREDfloe-guard1 ATTRIBUTE ANSWERED