A benchmark is a set of evaluation scenarios replayed automatically against a pipeline. It is the tool for objectively measuring agent quality and catching regressions before they reach production.
Why
An agent that "seemed to work" can silently regress after:
- a mission change;
- a LLM model change;
- a new tool being added.
Without a benchmark, you discover this in production. With a benchmark, you see it on the next run — before your users do.
Anatomy of a scenario
A scenario contains:
- A stimulus — the input the pipeline receives (a message or a sequence of turns, data fixtures).
- Checks — explicit assertions evaluated on each turn: the output produced, the shape of the response, the tool-call chain, the cost of actions.
Check catalogue
| Check | Verifies |
|---|---|
pipeline_output_match | A key of the output flow matches the expected value (path + assertion) |
response_shape | The shape of the agent's response (length, structure), measured without a judge |
llm_rubric | The substance of the response, scored by an LLM judge on a rubric of verifiable criteria |
tool_chain_analysis | The sequence of tool calls follows the expected strategy |
tool_cost | The action budget: each call has a cost, reaching the target earns a bonus |
static / tests / diff_quality | File- and code-oriented checks (output of a development pipeline) |
The catalogue is extensible platform-side: each check is a plugin registered on the server.
Creating a benchmark
- Applications → Evaluation → Scenarios.
- Create the scenarios (stimulus, checks, data fixtures); each scenario is dedicated to one model (the target pipeline). Fixtures are managed in the Fixtures tab.
A scenario can also be created or modified by proposal from the MCP server (benchmark_create / benchmark_update), which stays pending until it is accepted.
A good benchmark includes a mix: nominal case, edge cases, adversarial cases (manipulation attempts, ambiguity, missing data).
Running a benchmark
From the admin interface, launch a run: it is placed in a durable queue then executed by a dedicated worker, outside the web server. If this worker stops mid-run, the run is marked failed on restart (it is not resumed automatically): relaunch it. A drawer gives the per-scenario detail.
The report shows:
- Pass rate per check
- Score evolution across runs (History and Evolution tabs)
- List of failed scenarios, with the detail of each check
Best practices
- Start small — 5 well-targeted scenarios are worth more than 50 approximate ones.
- Benchmark before every publish, and compare with the previous run before pushing.
- Version your missions. Not just pipelines: missions evolve too.
- Keep checks readable. A check that requires 20 lines of explanation to understand is a check that needs to be broken down.