agent-observability-eval-bootstrap
agent-observability-eval-bootstrap is an agent skill published by datadog-labs, listed under deploy & operate. It lives in the datadog-labs/agent-skills repository, which has 178 GitHub stars and is licensed under MIT; we last read it on 2026-10-23.
Bootstrap evaluators from production traces by proposing online LLM-judge evaluators in Datadog as disabled drafts; also emit Python SDK code or a framework-agnostic JSON spec on request.
Install
skills is a generic installer for any repository of Claude agent skills -- it reads datadog-labs/agent-skills’s own structure rather than assuming one, the same way this page reads the skill’s own SKILL.md rather than assuming a format.
npx skills add datadog-labs/agent-skills
Maintenance
Bootstrap evaluators from production traces — by default propose online LLM-judge evaluators and, after you confirm, create them in Datadog as disabled drafts (never auto-enabled); on request emit Python SDK code or a framework-agnostic JSON spec instead. Use when user says "bootstrap evaluators", "generate evaluators", "create evals from traces", "eval bootstrap", "write evaluators", "build eval suite", "publish evaluators", or wants to generate BaseEvaluator/LLMJudge code or online judge configs from production LLM trace data. Works with ml_app and optional RCA report or failure hypothesis.
agent-observability-eval-bootstrap compared with similar skills
| Skill | Publisher | What it does | Repo stars | Last update |
|---|---|---|---|---|
| agent-observability-eval-bootstrap (this page) | datadog-labs | Bootstrap evaluators from production traces — by default propose online LLM-judge evaluators and, after you c… | 178 | 2026-10-22 |
| agent-observability-eval-pipeline(same repository) | datadog-labs | End-to-end Agent Observability pipeline for an instrumented ml_app — classify production traces, root-cause f… | 178 | 2026-10-22 |
| agent-observability-experiment-analyzer(same repository) | datadog-labs | Analyze LLM experiment results. Handles single or comparative experiments, exploratory or Q&A modes. Use when… | 178 | 2026-10-22 |
| agent-observability-trace-rca(same repository) | datadog-labs | Root cause analysis on production LLM traces. Diagnoses why an LLM application is failing — works from eval j… | 178 | 2026-10-22 |
| agent-skills(same repository) | datadog-labs | Datadog skills for AI agents. Essential monitoring, logging, tracing and observability. | 178 | 2026-10-22 |
| dd-aws-integration(same repository) | datadog-labs | Set up the Datadog AWS integration with Terraform - creates the cross-account IAM role Datadog assumes (exter… | 178 | 2026-10-22 |
| dd-azure-integration(same repository) | datadog-labs | Set up the Datadog Azure integration with Terraform - creates an Entra ID app registration and service princi… | 178 | 2026-10-22 |
Skills from the same repository first, then others for this job by repository stars. Figures are from each repository when we last read it.
Common questions about agent-observability-eval-bootstrap
- What does agent-observability-eval-bootstrap do?
- Bootstrap evaluators from production traces by proposing online LLM-judge evaluators in Datadog as disabled drafts; also emit Python SDK code or a framework-agnostic JSON spec on request.
- How do I install agent-observability-eval-bootstrap?
- Run npx skills add datadog-labs/agent-skills. The skills installer reads the repository’s own structure, so it works for any repository of agent skills.
- Who publishes agent-observability-eval-bootstrap?
- datadog-labs.
- Is agent-observability-eval-bootstrap maintained, and what licence does it use?
- Its repository has 178 GitHub stars and its state in our directory is listed. Its last commit we recorded was on 2026-10-22. The repository declares the MIT licence.
More agent skills for deploy & operate
- ai-content-pipelineby 101-skills
- explainer-video-guideby 101-skills
- ci-cd-and-automationby addyosmani
- observability-and-instrumentationby addyosmani
- internal-commsby anthropic
- xlsxby anthropic