Skip to content
InfratoApp
Layers / Observability / Evals / Evaluation and benchmarking

Evaluation and benchmarking

COMPARE TWO OF THESE →

Whether output is good, judged against a dataset or a rubric: scorers, regression suites, LLM-as-judge, human review. What happened at runtime is tracing and monitoring.

36 in this category
SORTED BY STARS · 4 UNMEASURED, GROUPED LAST

NARROW BY WHAT THIS CATEGORY ASKS

LICENCE · 8 NOT READ

OPEN SOURCE · 22 NOT READ

LLM-AS-JUDGE · 23 NOT READ

PRODUCTUSED BYSTARSDOWNLOADS 30D
langfuseLangfuse Python SDK - LLM observability/tracing, datasets, experiments, LLM-as-a-judge evaluation, and prompt management1634,9087,219,046PromptfooLLM eval & testing toolkit125,0052,424,795opikComet tool for logging and evaluating LLM traces122,182deepevalThe LLM Evaluation Framework18,230ragaai-catalystPython SDK for Agent AI Observability, Monitoring and Evaluation Framework. Includes features like agent, llm and tools tracing, debugging multi-agentic system, self-hosted dashboard and advanced anal16,164RagasEvaluation framework for RAG and LLM applications215,698ifixaiIndependent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have th15,663bishengBISHENG is an open LLM devops platform for next generation Enterprise AI applications. Powerful and comprehensive features include: GenAI workflow, RAG, Agent, Unified model management, Evaluation, SF11,993Arize PhoenixAI Observability and Evaluation411,4101,470,040AI-Infra-GuardA full-stack AI Red Teaming platform securing AI ecosystems via Agent Scan, Skills Scan, MCP scan, AI Infra scan and LLM jailbreak evaluation.6,536chinese-llm-benchmark非线智能 NoneLinear - ReLE评测：中文AI大模型能力评测（持续更新）：目前已囊括374个大模型，覆盖chatgpt、gpt-5.4、谷歌gemini-3.1-pro、Claude-4.6、文心ERNIE-X1.1、ERNIE-5.0、qwen3.6-max、qwen3.6-plus、百川、讯飞星火、商汤senseChat等商用模型， 以及step3.5-flash、kimi-k2.6,436giskard-oss🐢 Open-Source Evaluation & Testing library for LLM Agents5,811coze-loopNext-generation AI Agent Optimization Platform: Cozeloop addresses challenges in AI agent development by providing full-lifecycle management capabilities from development, debugging, and evaluation to5,716kilnBuild, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.5,058lmms-evalOne-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks4,399TruLensLibrary to systematically track and evaluate LLM based applications.3,54523,587langwatchThe platform for LLM evaluations and AI agent testing3,531MTEBMTEB: State-of-the-art evaluation of embeddings across languages and modalities3,420openlitOpen-source observability & evaluation platform for AI agents and coding agents. Trace LLMs, tools, prompts, costs & agent workflows with OpenTelemetry.2,756judgevalThe Continuous-Improvement Stack for Agents. Our environment data and evals power agent improvement and monitoring.1,060Ffinsight-aiAI equity research agent with resilient workflows, evidence-grounded RAG, versioned reports, and automated quality evaluation.1,004Aagent-skills-evalA test runner for agentskills.io-style AI agent skills731clawbenchOpen-source benchmark for browser AI agents on daily tasks.718EverMindOpen-source infrastructure that turns scattered SKILL.md files into curated, retrieval-ready agent-skill corpora—with retrieval and evaluation tooling included.646AdalineThe only fully local production-grade Super SDK that provides a simple, unified, and powerful interface for calling more than 200+ LLMs.606langtestDeliver safe & effective language models559PostTrainBenchMeasuring how well CLI agents like Claude Code or Codex CLI can post-train base LLMs on a single H100 GPU in 10 hours553continuous-evalData-Driven Evaluation for LLM-Powered Applications517Hharness-scoreYour AI coding agent is only as reliable as the harness around it. Measure that harness in seconds with harness-score.420AaaabenchGive a coding agent a brief, not a chat, and it works on its own across sessions. Includes an example run: an open-world city built in a real game engine with no human help. In early development.385AadarubricsAdaRubric: Adaptive Dynamic Rubric Evaluator for Agent Trajectories360BraintrustSDK for integrating Braintrust3275,607,117Fiddler AIPython client for Fiddler Platform24,780GalileoAI observability and evaluation for GenAI applicationsHoneyHiveHoneyHive Python SDK - LLM Observability and Evaluation Platform12,994LangSmithClient library to connect to the LangSmith Observability and Evaluation Platform.1524,619,866

SORTED BY ONE STATED COLUMN · USED BY = LISTED PRODUCTS DEPENDING ON IT, IN THIS DIRECTORY — NOT A POPULARITY FIGURE · NO OVERALL SCORE AND NO RANK COLUMN, BECAUSE BLENDING ADOPTION, STARS AND DOWNLOADS NEEDS WEIGHTS, AND WEIGHTS ARE AN OPINION PRESENTED AS ARITHMETIC

Cite this page

The count and its date sit inside the citation, because “Evaluation and benchmarking” without a date is a claim that quietly goes wrong as the category fills.

infratoapp, “Evaluation and benchmarking — Observability / Evals”, read 27 October 2026. 36 listed products. https://www.infratoapp.com/layers/observability/evaluation

No overall score, and no rank column

Blending adoption, stars and downloads needs weights, and weights are an opinion presented as arithmetic — so this table sorts by one stated column at a time and shows which. This view holds no billing column and joins no billing table: a sponsor cannot appear higher, and the signals meter is a row’s own confidence, never a score you can sort on.

Frequently asked

What is in the Evaluation and benchmarking category?
36 products in the Observability / Evals layer of the AI stack. Whether output is good, judged against a dataset or a rubric: scorers, regression suites, LLM-as-judge, human review. What happened at runtime is tracing and monitoring. Listed members include langfuse, Promptfoo and opik. Every fact on each listing carries the page it was read from and the date it was read.
How are Evaluation and benchmarking compared here?
By the 7 attributes this category declares, asked of every member and answered only where a source states it. A product with no answer on record shows a gap rather than a guess, and no blended score is published — weights would be an opinion presented as arithmetic.
langfuse or Promptfoo?
Both are listed under Evaluation and benchmarking. We publish no verdict between them — we hold no benchmark that would support one. What we do hold is the same attribute sheet for each, side by side with the source behind every value, at /compare/langfuse-vs-promptfoo.
What does a blank mean on a evaluation and benchmarking row?
That nobody has told us, not that the answer is no. A hollow ring is "no answer on record", which is a different state from a recorded no — and the two look different on purpose. A listing with a blank has not failed the question; it has not been read for it.
How often are evaluation and benchmarking figures re-read?
Every value carries the date it was read and a link to the page it came from. Prices move fastest and are re-read most often; an attribute read weeks ago says so rather than presenting itself as current. Anything past 21 days is hidden rather than shown stale.
Is this every evaluation and benchmarking there is?
No. This is 36 we have read and can cite. A directory that claimed completeness would be making a promise it cannot keep, so the page states its own count instead and says when a category is too thin to shortlist from.
Can a vendor pay to rank higher here?
No. Nothing on this page can be bought: a sponsor cannot enter a table, reorder a row or change a mark, and there is no ranked position for sale. How the site is paid for, and the line each source of money does not cross, is set out under money and independence at /our-data#sec5.