agent-observability-build-eval-from-annotations
agent-observability-build-eval-from-annotations is an agent skill published by datadog-labs, listed under deploy & operate. It lives in the datadog-labs/agent-skills repository, which has 178 GitHub stars and is licensed under MIT; we last read it on 2026-09-25.
Fit a Datadog LLM-Obs evaluator to human labels by analyzing an annotation queue, drafting and optimizing a judge to predict labels, and publishing the best model as a disabled evaluator in Datadog. Requires at least two classes in the human labels.
Install
skills is a generic installer for any repository of Claude agent skills -- it reads datadog-labs/agent-skills’s own structure rather than assuming one, the same way this page reads the skill’s own SKILL.md rather than assuming a format.
npx skills add datadog-labs/agent-skills
Maintenance
Fit a Datadog LLM-Obs evaluator to human labels. Takes an annotation queue, works out where in the trace the labelled property actually lives, drafts an LLM-judge that predicts the human label, scores that judge against the already-labelled rows with a metric agreed with the user, then hill-climbs it — inspect the errors, make one focused change, re-score, keep it only if it beats the best — for a bounded number of iterations, and finally publishes the winner to Datadog as a DISABLED evaluator (not a Datadog draft — a real evaluator with `enabled: false`). Use when the user says "build an eval from my annotations", "build an evaluator from the annotation queue", "turn my annotations into an evaluator", "learn an evaluator from my labels", "fit a judge to the annotation queue", "auto-label", "auto labelling", "automate this annotation queue", "scale up my human labels", or wants the rest of a queue graded the way the humans graded the first rows. Needs an annotation queue with at least two classes present in the human labels (e.g. one true and one false for a boolean).
agent-observability-build-eval-from-annotations compared with similar skills
| Skill | Publisher | What it does | Repo stars | Last update |
|---|---|---|---|---|
| agent-observability-build-eval-from-annotations (this page) | datadog-labs | Fit a Datadog LLM-Obs evaluator to human labels. Takes an annotation queue, works out where in the trace the… | 178 | 2026-10-22 |
| agent-observability-eval-bootstrap(same repository) | datadog-labs | Bootstrap evaluators from production traces — by default propose online LLM-judge evaluators and, after you c… | 178 | 2026-10-22 |
| agent-observability-eval-pipeline(same repository) | datadog-labs | End-to-end Agent Observability pipeline for an instrumented ml_app — classify production traces, root-cause f… | 178 | 2026-10-22 |
| agent-observability-experiment-analyzer(same repository) | datadog-labs | Analyze LLM experiment results. Handles single or comparative experiments, exploratory or Q&A modes. Use when… | 178 | 2026-10-22 |
| agent-observability-trace-rca(same repository) | datadog-labs | Root cause analysis on production LLM traces. Diagnoses why an LLM application is failing — works from eval j… | 178 | 2026-10-22 |
| agent-skills(same repository) | datadog-labs | Datadog skills for AI agents. Essential monitoring, logging, tracing and observability. | 178 | 2026-10-22 |
| dd-aws-integration(same repository) | datadog-labs | Set up the Datadog AWS integration with Terraform - creates the cross-account IAM role Datadog assumes (exter… | 178 | 2026-10-22 |
Skills from the same repository first, then others for this job by repository stars. Figures are from each repository when we last read it.
Common questions about agent-observability-build-eval-from-annotations
- What does agent-observability-build-eval-from-annotations do?
- Fit a Datadog LLM-Obs evaluator to human labels by analyzing an annotation queue, drafting and optimizing a judge to predict labels, and publishing the best model as a disabled evaluator in Datadog. Requires at least two classes in the human labels.
- How do I install agent-observability-build-eval-from-annotations?
- Run npx skills add datadog-labs/agent-skills. The skills installer reads the repository’s own structure, so it works for any repository of agent skills.
- Who publishes agent-observability-build-eval-from-annotations?
- datadog-labs.
- Is agent-observability-build-eval-from-annotations maintained, and what licence does it use?
- Its repository has 178 GitHub stars and its state in our directory is listed. Its last commit we recorded was on 2026-10-22. The repository declares the MIT licence.
More agent skills for deploy & operate
- ai-content-pipelineby 101-skills
- explainer-video-guideby 101-skills
- ci-cd-and-automationby addyosmani
- observability-and-instrumentationby addyosmani
- internal-commsby anthropic
- xlsxby anthropic