agent-observability-auto-experiment
agent-observability-auto-experiment is an agent skill published by datadog-labs, listed under deploy & operate. It lives in the datadog-labs/agent-skills repository, which has 178 GitHub stars and is licensed under MIT; we last read it on 2026-09-25.
Runs iterative code improvements against real Datadog LLM-Obs data using Claude Code, establishing a baseline evaluation, making focused changes, and repeating if improvements are noted. Requires a Datadog LLM-Obs dataset, annotation queue ID, and trace IDs.
Install
skills is a generic installer for any repository of Claude agent skills -- it reads datadog-labs/agent-skills’s own structure rather than assuming one, the same way this page reads the skill’s own SKILL.md rather than assuming a format.
npx skills add datadog-labs/agent-skills
What it connects to
Unverified means a mention in the skill’s own description, not a citation. A wrong claim here costs nothing to correct and nothing to trust yet.
Maintenance
Run an iterative code-improvement hill-climb against real Datadog LLM-Obs data, locally, with Claude Code as the agent. Establishes a baseline eval, makes one focused change, re-scores with the same harness, keeps the change if it improves the score in the goal's direction (labeling within-noise gains tentative), and repeats. Use when the user says "run an auto experiment", "hill-climb this code", "iteratively improve X and measure the delta", "optimize this prompt/file against my traces", "auto-optimize against LLM-Obs", or wants the local equivalent of the auto_experiments worker. Works from an ml_app, a dataset_id, an annotation_queue_id (a queue of human-labelled interactions), a list of trace_ids, or (by exception) a local dataset file. The corpus and its val/test splits live in Datadog LLM-Obs Datasets, created once per run with a timestamp in their names.
agent-observability-auto-experiment compared with similar skills
| Skill | Publisher | What it does | Repo stars | Last update |
|---|---|---|---|---|
| agent-observability-auto-experiment (this page) | datadog-labs | Run an iterative code-improvement hill-climb against real Datadog LLM-Obs data, locally, with Claude Code as… | 178 | 2026-10-22 |
| agent-observability-eval-bootstrap(same repository) | datadog-labs | Bootstrap evaluators from production traces — by default propose online LLM-judge evaluators and, after you c… | 178 | 2026-10-22 |
| agent-observability-eval-pipeline(same repository) | datadog-labs | End-to-end Agent Observability pipeline for an instrumented ml_app — classify production traces, root-cause f… | 178 | 2026-10-22 |
| agent-observability-experiment-analyzer(same repository) | datadog-labs | Analyze LLM experiment results. Handles single or comparative experiments, exploratory or Q&A modes. Use when… | 178 | 2026-10-22 |
| agent-observability-trace-rca(same repository) | datadog-labs | Root cause analysis on production LLM traces. Diagnoses why an LLM application is failing — works from eval j… | 178 | 2026-10-22 |
| agent-skills(same repository) | datadog-labs | Datadog skills for AI agents. Essential monitoring, logging, tracing and observability. | 178 | 2026-10-22 |
| dd-aws-integration(same repository) | datadog-labs | Set up the Datadog AWS integration with Terraform - creates the cross-account IAM role Datadog assumes (exter… | 178 | 2026-10-22 |
Skills from the same repository first, then others for this job by repository stars. Figures are from each repository when we last read it.
Common questions about agent-observability-auto-experiment
- What does agent-observability-auto-experiment do?
- Runs iterative code improvements against real Datadog LLM-Obs data using Claude Code, establishing a baseline evaluation, making focused changes, and repeating if improvements are noted. Requires a Datadog LLM-Obs dataset, annotation queue ID, and trace IDs.
- How do I install agent-observability-auto-experiment?
- Run npx skills add datadog-labs/agent-skills. The skills installer reads the repository’s own structure, so it works for any repository of agent skills.
- Which AI clients can use agent-observability-auto-experiment?
- Its own description mentions Claude Code. That is a mention in its description, not a test we ran.
- Who publishes agent-observability-auto-experiment?
- datadog-labs.
- Is agent-observability-auto-experiment maintained, and what licence does it use?
- Its repository has 178 GitHub stars and its state in our directory is listed. Its last commit we recorded was on 2026-10-22. The repository declares the MIT licence.
More agent skills for deploy & operate
- ai-content-pipelineby 101-skills
- explainer-video-guideby 101-skills
- ci-cd-and-automationby addyosmani
- observability-and-instrumentationby addyosmani
- internal-commsby anthropic
- xlsxby anthropic