creating-online-evaluations
creating-online-evaluations is the official agent skill of PostHog published by posthog, listed under analytics & experiments. It lives in the PostHog/skills repository, which has 75 GitHub stars and is licensed under MIT; we last read it on 2026-10-22.
Automatically create and manage continuous online evaluations in PostHog AI observability based on identified failure modes. Requires a provider key for certain eval types.
Install
skills is a generic installer for any repository of Claude agent skills -- it reads PostHog/skills’s own structure rather than assuming one, the same way this page reads the skill’s own SKILL.md rather than assuming a format.
npx skills add PostHog/skills
What it connects to
Unverified means a mention in the skill’s own description, not a citation. A wrong claim here costs nothing to correct and nothing to trust yet.
Maintenance
Author continuously-running online evaluations in PostHog AI observability, grounded in real failure modes you've identified. Use when the user wants evaluations that automatically score new generations or whole traces going forward — "create an eval to catch X", "continuously check that responses do Y", "turn these failures into evals". Covers letting the explored data decide how many evals to create, proposing that set for the user to pick, choosing the target and eval type (hog / llm_judge / sentiment), configuring a provider and model for an llm_judge eval (a provider key gates enabling, not creation), scoping which generations trigger it via conditions, creating disabled, verifying scope, enabling, and backfilling past data. Proposes a sentiment eval when no failure mode is worth catching. Finding and ranking the failure modes worth evaluating is its own job — use exploring-ai-failures first. To debug or manage evaluations that already exist, use exploring-llm-evaluations.
creating-online-evaluations compared with similar skills
| Skill | Publisher | What it does | Repo stars | Last update |
|---|---|---|---|---|
| creating-online-evaluations (this page) | posthog | Author continuously-running online evaluations in PostHog AI observability, grounded in real failure modes yo… | 75 | 2026-10-25 |
| account-handover(same repository) | posthog | Draft structured handover notes for transitioning a PostHog account from one TAM or CSM to another. Use this… | 75 | 2026-10-25 |
| adding-warehouse-person-properties(same repository) | posthog | Sync columns from a synced data warehouse table onto PostHog person or group properties, so warehouse data be… | 75 | 2026-10-25 |
| analyzing-experiment-session-replays(same repository) | posthog | Analyze session replay patterns across experiment variants to understand user behavior differences. Use when… | 75 | 2026-10-25 |
| analyzing-task-runs(same repository) | posthog | Split a completed PostHog task run into activity records — what the agent tried, whether it worked, what bloc… | 75 | 2026-10-25 |
| assessing-heatmaps(same repository) | posthog | Assesses what a page's heatmap is telling you and recommends concrete changes. Pulls click / rageclick / scro… | 75 | 2026-10-25 |
| auditing-endpoints(same repository) | posthog | Audit every endpoint in a PostHog project for staleness, failed materialisations, and unused materialised ver… | 75 | 2026-10-25 |
Skills from the same repository first, then others for this job by repository stars. Figures are from each repository when we last read it.
Common questions about creating-online-evaluations
- What does creating-online-evaluations do?
- Automatically create and manage continuous online evaluations in PostHog AI observability based on identified failure modes. Requires a provider key for certain eval types.
- How do I install creating-online-evaluations?
- Run npx skills add PostHog/skills. The skills installer reads the repository’s own structure, so it works for any repository of agent skills.
- Who publishes creating-online-evaluations?
- posthog. It is named as the official skill of PostHog.
- Is creating-online-evaluations maintained, and what licence does it use?
- Its repository has 75 GitHub stars and its state in our directory is listed. Its last commit we recorded was on 2026-10-25. The repository declares the MIT licence.
More agent skills for analytics & experiments
- youtube-thumbnail-designby 101-skills
- incremental-implementationby addyosmani
- agent-observability-replay-traceby datadog-labs
- dd-browser-sdkby datadog-labs
- dd-instrument-rumby datadog-labs
- omni-ai-evalby exploreomni