> For the complete documentation index, see [llms.txt](/llms.txt). > A full single-fetch corpus is available at [llms-full.txt](/llms-full.txt). --- title: Experiments description: Run a deployed component against a dataset version, score every item, compare runs, and gate CI on the results. last_verified: 2026-10-08 --- An **experiment** binds a target (a deployed component or a prompt) to a [dataset](/docs/improve/datasets.md) version and a set of [scorers](/docs/improve/scorers.md). Each **experiment run** executes the target against every dataset item, scores the outputs, and produces a pass/fail summary you can compare across runs or enforce in CI. --- ## Experiment versions - An experiment's definition (target, scorers, config) is versioned. Starting a run snapshots it as an immutable **experiment version**, so older runs always show the exact configuration that produced them. - A run pairs one experiment version with one dataset version. Because both sides are immutable, two runs over the same dataset version are directly comparable. - Each dataset item becomes a **run item** with its own output and per-scorer scores. --- ## List experiments ```bash # All experiments in the current project agnt5 experiments list # Filter by dataset, status, or type agnt5 experiments list --dataset-id --status draft ``` Available filters: `--dataset-id`, `--status`, `--type`. Use `--page` and `--page-size` for pagination. --- ## Create an experiment The target component must already be [deployed](/docs/run/deploying.md). Then: ```bash agnt5 experiments create \ --name support-agent-quality \ --dataset-id \ --dataset-version-id \ --target-type component \ --deployment-id \ --component-name support_agent \ --component-type agent \ --builtin-scorer json_valid \ --builtin-scorer '{"name":"tool_called","config":{"tool":"search_orders"}}' \ --config '{"passed_threshold":1}' ``` - `--builtin-scorer` attaches a [built-in scorer](/docs/improve/scorers.md#built-in-deterministic-scorers) by name, or by JSON object when it needs config. Repeat for each scorer. - `--scorer-id` attaches a deployed [custom scorer](/docs/improve/scorers.md#custom-scorers) by the ID of its scorer record ([Create a scorer](/docs/api-reference/scorers/create-a-scorer.md)). - `--config '{"passed_threshold":0.5}'` sets the pass rate the run must reach for its gate to pass. - `--target-type prompt` with `--prompt-id` targets a [Prompt](/docs/build/prompts.md) instead of a component: ```bash agnt5 experiments create \ --name support-prompt-comparison \ --dataset-id \ --dataset-version-id \ --target-type prompt \ --deployment-id \ --prompt-id \ --prompt-version-id \ --builtin-scorer correctness ``` Via Studio: open your project, go to **Evaluate** -> **Experiments**, and create the experiment with the same choices — dataset version, target, and scorers. --- ## Run an experiment ```bash # Fire and forget agnt5 experiments run # Block until the run finishes; exit non-zero if the gate fails agnt5 experiments run --wait --timeout 15m ``` With `--wait`, the output shows the run's pass rate against the experiment's threshold: ``` RUN STATUS PASSED PASS_RATE THRESHOLD PROGRESS FAILED 3f7a1c20.. completed true 66.7% 50.0% 3/3 0 ``` Pass `--name` to label the run, `--deployment-id` to run a candidate deployment, or `--experiment-version-id` to re-run an older experiment version. See [Eval commands](/docs/cli/evals.md#agnt5-experiments) for every flag. --- ## Inspect results ```bash # Runs for an experiment agnt5 experiments runs list # One run's summary: status, pass rate, per-scorer aggregates agnt5 experiments runs show # Also available as a standalone command: agnt5 reports summary # Failed items only agnt5 reports failures # Scores for a run — filter by scorer, item, component, or time window agnt5 scores list --run-id agnt5 scores list --run-id --scorer-id agnt5 scores list --run-id --run-item-id agnt5 scores list --since 2h --component-name support_agent # Hydrate scorer input, output, and evidence for one score agnt5 scores evidence --include scorer_input,scorer_output,evidence ``` In Studio, the experiment run page shows the run timeline, per-item results, and per-scorer scores; click any item to see its output and score evidence side by side. Cancel a stuck run with `agnt5 experiments runs cancel `. --- ## Compare runs Compare a candidate against a baseline over the same dataset version: ```bash agnt5 experiments runs compare ``` The comparison shows the pass-rate and failed-item deltas, and the per-scorer movement in count, passed, failed, and average score. Studio renders the same comparison under **Evaluate** -> **Experiments** when you select two runs. --- ## Gate CI on eval results Use the reports commands in a CI job to block a merge or deploy on eval regressions: ```bash # Start a run for the freshly deployed candidate, wait, and gate agnt5 experiments run \ --deployment-id "$CANDIDATE_DEPLOYMENT_ID" \ --name "ci-$GIT_SHA" \ --wait --fail-on-gate # Or wait on an already-started run agnt5 reports wait --timeout 15m # Print the gate verdict for a finished run agnt5 reports ci # Export full artifacts for the build log agnt5 reports export --artifact-format csv --out-file eval-results.csv ``` A failed gate exits with code `2`. See [Exit codes](/docs/cli/evals.md#exit-codes) for the others. Use `--fail-on-gate=false` to suppress non-zero exits and inspect the result yourself. The gate verdict comes from the experiment's config (for example `{"passed_threshold": 1}`), so the threshold lives with the experiment, not the pipeline. --- ## Turn failures into a regression dataset After a run with failures, capture the failing items as a new dataset so fixes stay fixed: ```bash agnt5 experiments runs regression-dataset \ --name support-agent-regressions \ --start-run --wait # Rerun against a specific deployment (e.g. a fix candidate) agnt5 experiments runs regression-dataset \ --name support-agent-regressions \ --start-run --deployment-id --wait ``` This creates a dataset from the failed items (all of them, or specific ones via repeatable `--run-item-id`), a regression experiment over it, and — with `--start-run` — kicks off the first rerun immediately. Pass `--deployment-id` to run the regression against a specific deployment rather than the experiment's default. --- ## Rescore a completed run After updating a scorer's logic or thresholds, rescore an existing completed run without re-executing the component. The original outputs stay intact; only the score records are replaced. Call the API with a personal API key (see [Authentication](/docs/api-reference/authentication.md)): ```bash curl -X POST "https://api.agnt5.com/api/v1/projects//eval/runs//rescore" \ -H "X-API-KEY: " \ -H "Content-Type: application/json" \ -d '{"reason": "updated_rubric", "replace_latest": true}' ``` The response counts the scoring jobs it queued, one per item and scorer. Narrow the rescore with `run_item_ids` or `scorer_version_ids`, and send an `idempotency_key` to read the job counts again without queuing them twice. See [Rescore a run](/docs/api-reference/experiments/rescore-a-run.md) for every field. Run the gate command after the jobs finish to see the updated verdict: ```bash agnt5 reports ci ``` --- ## Annotate run items Add a human label to any run item to capture reviewer decisions alongside scorer verdicts. Annotations are stored separately from scores and used for meta-evaluation (measuring scorer accuracy against human judgement). Call the API with a personal API key (see [Authentication](/docs/api-reference/authentication.md)). Get the item ID from `agnt5 reports failures --format jsonl` or the run's items. ```bash curl -X POST "https://api.agnt5.com/api/v1/projects//eval/runs//items//annotations" \ -H "X-API-KEY: " \ -H "Content-Type: application/json" \ -d '{"label": "pass", "metadata": {"reviewer": "alice", "note": "Correct order ID cited"}}' ``` `label` is required; `name` defaults to `human_label`. See [Create an annotation](/docs/api-reference/experiments/create-an-annotation.md) and [List annotations](/docs/api-reference/experiments/list-annotations.md) for the full reference. In Studio, annotations appear inline next to scorer results in the run item detail view. ## Output formats Eval commands print tables on a terminal. Add `--format json` or `--format jsonl` for scripting, and `--project-id` to target another project. See [Output formats and project targeting](/docs/cli/evals.md#output-formats-and-project-targeting). --- **Commands**: `agnt5 experiments {list,create,run,runs list|show|cancel|compare|regression-dataset}`, `agnt5 reports {summary,failures,ci,wait,export}`, `agnt5 scores {list,evidence}` **Create requires**: `--name`, `--dataset-id`, `--dataset-version-id`, target (`--target-type component --deployment-id --component-name --component-type`, or `--target-type prompt --prompt-id`), at least one `--builtin-scorer ` or `--scorer-id ` **CI gating**: `experiments run --wait --fail-on-gate` and `reports ci|wait` exit non-zero on gate failure; threshold set via experiment `--config '{"passed_threshold": <0..1>}'`; exit codes: 2=gate failed, 3=run failed/cancelled, 4=timeout **Rescore**: `POST /api/v1/projects/{projectId}/eval/runs/{runId}/rescore` (personal API key) — completed runs; optional `scorer_version_ids` and `run_item_ids` to narrow scope; `replace_latest: true` refreshes aggregates; `idempotency_key` re-reads progress. **Annotations**: `POST|GET /api/v1/projects/{projectId}/eval/runs/{runId}/items/{runItemId}/annotations` with `label` (required), `name`, `metadata`; used for meta-evaluation against human judgement. Auth: personal API key in `X-API-KEY`. **Model**: experiment versions and dataset versions are immutable; a run = one experiment version × one dataset version; per-item scores queryable via `scores list --run-id`. ## Next steps * [Datasets](/docs/improve/datasets.md): grow the dataset behind the experiment and publish a new version. * [Scorers](/docs/improve/scorers.md): tighten what pass/fail means, or write a custom scorer. * [Quality cases](/docs/improve/quality-cases.md): track regressions and production failures through a structured lifecycle. * [Deploying](/docs/run/deploying.md): ship the candidate deployment your experiment runs against. * [Prompts](/docs/build/prompts.md): version the prompt artifacts that prompt-target experiments compare.