> For the complete documentation index, see [llms.txt](/llms.txt).
> A full single-fetch corpus is available at [llms-full.txt](/llms-full.txt).
---
title: Eval commands
description: Reference for managing eval datasets, experiments, reports, and scores from the CLI.
category: CLI
last_verified: 2026-07-24
hideRightSidebar: true
---

Four command groups cover the eval lifecycle: **`agnt5 datasets`** curates test data, **`agnt5 experiments`** runs a component or prompt against a dataset version, **`agnt5 reports`** summarizes a run's results, and **`agnt5 scores`** queries individual scorer outputs. All four accept `--project-id` (defaults to the current project) and `--format table|json|jsonl` (defaults to the global `--output`).


**Commands**: `agnt5 datasets <cmd>`, `agnt5 experiments <cmd>`, `agnt5 reports <cmd>`, `agnt5 scores <cmd>`
**Shared flags**: `--project-id`, `--format`
**Typical flow**: `datasets create` → `datasets upload`/`add-run` → `datasets publish` → `experiments create` → `experiments run` → `reports summary`/`reports ci` → `scores list`
**Side effects**: `datasets publish` creates an immutable version; `experiments run` and `datasets dedup apply` mutate remote state
**Required**: `--name` and `--dataset-id` on `experiments create`; `datasets versions compare` takes version UUIDs (from `versions list`), not version numbers


## Command summary

| Group | Purpose |
| --- | --- |
| `datasets` | Create, curate, deduplicate, and version eval datasets. |
| `experiments` | Configure and run components/prompts against a dataset. |
| `reports` | Summarize, export, and gate CI on an experiment run's results. |
| `scores` | Query individual scorer outputs and their evidence. |

## `agnt5 datasets`

```bash
agnt5 datasets <command> [options]
```

| Command | Purpose |
| --- | --- |
| `list` | List eval datasets. |
| `create` | Create a dataset. |
| `add-run <dataset-id> <run-id>` | Import a runtime run into the dataset draft. |
| `add-example <dataset-id>` | Add a manual example to the draft. |
| `examples list <dataset-id>` | List examples in the draft or a version. |
| `upload <dataset-id>` | Upload JSONL examples into the draft. |
| `upload-csv <dataset-id>` | Upload CSV examples with column mapping. |
| `dedup preview <dataset-id>` | Preview duplicate draft examples. |
| `dedup apply <dataset-id>` | Remove duplicate draft examples. |
| `publish <dataset-id>` | Save the draft as an immutable version. |
| `versions list <dataset-id>` | List a dataset's versions. |
| `versions export <dataset-id> <version-id>` | Export a version as JSONL. |
| `versions compare <dataset-id> <base-version-id> <compare-version-id>` | Compare two versions, by version ID (not version number). |
| `versions restore-draft <dataset-id> <version-id>` | Restore the draft from a version. |

#### `datasets list`

| Flag | Description |
| --- | --- |
| `--page <n>` | Page number. Default: `1`. |
| `--page-size <n>` | Results per page. Default: `20`. |
| `--search <text>` | Filter datasets by name or description. |

#### `datasets create`

| Flag | Description |
| --- | --- |
| `--name <name>` | Dataset name. |
| `--description <text>` | Dataset description. |
| `--metadata <json>` | Arbitrary metadata as a JSON object. |

#### `datasets add-run <dataset-id> <run-id>`

| Flag | Description |
| --- | --- |
| `--expected-output <json>` | Expected output for the imported example. |
| `--metadata <json>` | Arbitrary metadata as a JSON object. |

#### `datasets add-example <dataset-id>`

| Flag | Description |
| --- | --- |
| `--input <json>` | Example input. |
| `--expected-output <json>` | Expected output. |
| `--metadata <json>` | Arbitrary metadata as a JSON object. |

#### `datasets examples list <dataset-id>`

| Flag | Description |
| --- | --- |
| `--page <n>` | Page number. Default: `1`. |
| `--page-size <n>` | Results per page. Default: `30`. |
| `--version <n>` | Immutable version number to list. Omit for the draft. |
| `--include-payload` | Include full input/output payloads in the output. |

#### `datasets upload <dataset-id>`

| Flag | Description |
| --- | --- |
| `--file <path>`, `-f` | JSONL file to upload, or `-` for stdin. Default: `-`. |

#### `datasets upload-csv <dataset-id>`

| Flag | Description |
| --- | --- |
| `--file <path>`, `-f` | CSV file to upload, or `-` for stdin. Default: `-`. |
| `--input-column <name>` | Column mapped to the example input. Default: `input`. |
| `--expected-output-column <name>` | Column mapped to the expected output. Default: `expected_output`. |
| `--metadata-column <name>` | Column mapped to example metadata. |
| `--events-column <name>` | Column mapped to recorded events. |
| `--split-column <name>` | Column mapped to the dataset split. |
| `--tags-column <name>` | Column mapped to example tags. |
| `--source-run-id-column <name>` | Column mapped to the source run ID. |
| `--source-ref-column <name>` | Column mapped to the source reference. |
| `--delimiter <char>` | Field delimiter. Default: `,`. |
| `--no-header` | Treat the first row as data, not a header. |
| `--partial` | Continue past rows that fail to parse. Default: `true`. |
| `--source-ref <ref>` | Source reference applied to every imported example. |

#### `datasets dedup preview <dataset-id>`

| Flag | Description |
| --- | --- |
| `--include-preview` | Include the duplicate example payloads in the preview. |

#### `datasets dedup apply <dataset-id>`

| Flag | Description |
| --- | --- |
| `--remove-item-id <uuid>` | Item to remove; repeatable. Omit to keep the earliest item in each duplicate group. |

#### `datasets publish <dataset-id>`

| Flag | Description |
| --- | --- |
| `--description <text>` | Description for the new version. |
| `--metadata <json>` | Arbitrary metadata as a JSON object. |

#### `datasets versions list <dataset-id>`

| Flag | Description |
| --- | --- |
| `--page <n>` | Page number. Default: `1`. |
| `--page-size <n>` | Results per page. Default: `20`. |

`versions export`, `versions compare`, and `versions restore-draft` take only positional arguments. `versions compare` sends its `<base-version-id>`/`<compare-version-id>` arguments straight through as version UUIDs — get them from `agnt5 datasets versions list <dataset-id>` first, not the version number shown there.

```bash
agnt5 datasets create --name support-regressions --description "Failed support runs"
agnt5 datasets add-run ds_a1 run_9f2a --expected-output '{"resolved": true}'
agnt5 datasets upload ds_a1 --file examples.jsonl
agnt5 datasets upload-csv ds_a1 --file examples.csv --input-column prompt --expected-output-column answer
agnt5 datasets dedup preview ds_a1
agnt5 datasets dedup apply ds_a1
agnt5 datasets publish ds_a1 --description "v3 with dedup applied"
agnt5 datasets versions list ds_a1
agnt5 datasets versions compare ds_a1 7f6e5d4c-9b8a-4c3d-a1e2-f0b1c2d3e4f5 2a1b3c4d-5e6f-4a7b-8c9d-0e1f2a3b4c5d
```

## `agnt5 experiments`

```bash
agnt5 experiments <command> [options]
```

| Command | Purpose |
| --- | --- |
| `list` | List experiments. |
| `create` | Create an experiment. |
| `run <experiment-id>` | Start a run. |
| `runs list <experiment-id>` | List runs for an experiment. |
| `runs show <run-id>` | Show a run summary. |
| `runs cancel <experiment-id> <run-id>` | Cancel a running run. |
| `runs regression-dataset <run-id>` | Create a regression dataset from failed items. |
| `runs compare <base-run-id> <compare-run-id>` | Compare two runs. |

#### `experiments list`

| Flag | Description |
| --- | --- |
| `--page <n>` | Page number. Default: `1`. |
| `--page-size <n>` | Results per page. Default: `20`. |
| `--dataset-id <id>` | Filter by dataset. |
| `--status <status>` | Filter by run status. |
| `--type <type>` | Filter by experiment type. |

#### `experiments create`

| Flag | Description |
| --- | --- |
| `--name <name>` | Experiment name. **Required.** |
| `--description <text>` | Experiment description. |
| `--dataset-id <id>` | Dataset to evaluate against. **Required.** |
| `--dataset-version-id <id>` | Pin a specific dataset version. |
| `--type <type>` | Experiment type. Default: `evaluation`. |
| `--target-type <type>` | Target to run: `component` (default), `deployment`, or `prompt`. |
| `--deployment-id <id>` | Deployment to target when `--target-type deployment`. |
| `--component-name <name>` | Component to target when `--target-type component`. |
| `--component-type <type>` | Component type. Default: `function`. |
| `--prompt-id <id>` | Prompt to target when `--target-type prompt`. |
| `--prompt-version-id <id>` | Pin a specific prompt version. |
| `--prompt-selection-mode <mode>` | How the prompt version resolves. |
| `--scorer-id <uuid>` | Custom scorer to attach; repeatable. |
| `--builtin-scorer <name-or-json>` | Built-in scorer to attach; repeatable. |
| `--config <json>` | Experiment configuration as a JSON object. |

#### `experiments run <experiment-id>`

| Flag | Description |
| --- | --- |
| `--name <name>` | Run name. |
| `--deployment-id <id>` | Deployment to run against. |
| `--experiment-version-id <id>` | Pin a specific experiment version. |
| `--config <json>` | Run configuration as a JSON object. |
| `--wait` | Block until the run finishes. |
| `--timeout <duration>` | Max time to wait with `--wait`. Default: `10m`. |
| `--fail-on-gate` | Exit non-zero when the CI gate fails. Default: `true`. |

#### `experiments runs list <experiment-id>`

| Flag | Description |
| --- | --- |
| `--page <n>` | Page number. Default: `1`. |
| `--page-size <n>` | Results per page. Default: `20`. |
| `--status <status>` | Filter by run status. |

#### `experiments runs regression-dataset <run-id>`

| Flag | Description |
| --- | --- |
| `--name <name>` | Name for the new regression dataset. |
| `--description <text>` | Dataset description. |
| `--metadata <json>` | Dataset metadata as a JSON object. |
| `--version-description <text>` | Description for the published version. |
| `--version-metadata <json>` | Version metadata as a JSON object. |
| `--experiment-name <name>` | Name for a new experiment over the dataset. |
| `--experiment-description <text>` | Description for the new experiment. |
| `--run-item-id <uuid>` | Failed item to export; repeatable. Omit to export every failed item. |
| `--start-run` | Start a run against the new dataset immediately. |
| `--run-name <name>` | Name for the started run. |
| `--deployment-id <id>` | Deployment to run against. |
| `--wait` | Block until the started run finishes. |
| `--timeout <duration>` | Max time to wait with `--wait`. Default: `10m`. |
| `--fail-on-gate` | Exit non-zero when the CI gate fails. Default: `false`. |

`runs show`, `runs cancel`, and `runs compare` take only positional arguments.

```bash
agnt5 experiments create --name weather-agent-v2 --dataset-id ds_a1 \
  --target-type component --component-name weather-agent --component-type agent \
  --builtin-scorer correctness
agnt5 experiments run exp_7c1e --wait --fail-on-gate
agnt5 experiments runs list exp_7c1e --status failed
agnt5 experiments runs regression-dataset run_9f2a --name "weather-agent-regressions" --start-run
agnt5 experiments runs compare run_base run_compare
```

## `agnt5 reports`

```bash
agnt5 reports <command> <run-id> [options]
```

| Command | Purpose |
| --- | --- |
| `summary <run-id>` | Show a run summary. |
| `failures <run-id>` | List failed examples. |
| `ci <run-id>` | Show the CI gate result. |
| `wait <run-id>` | Wait for a run to finish, then print its CI gate result. |
| `export <run-id>` | Export run artifacts. |

#### `reports failures <run-id>`

| Flag | Description |
| --- | --- |
| `--page <n>` | Page number. Default: `1`. |
| `--page-size <n>` | Results per page. Default: `20`. |

#### `reports ci <run-id>`

| Flag | Description |
| --- | --- |
| `--fail-on-gate` | Exit non-zero when the CI gate fails. Default: `true`. |

#### `reports wait <run-id>`

| Flag | Description |
| --- | --- |
| `--timeout <duration>` | Max time to wait for the run to finish. Default: `10m`. |
| `--fail-on-gate` | Exit non-zero when the CI gate fails. Default: `true`. |

#### `reports export <run-id>`

| Flag | Description |
| --- | --- |
| `--artifact-format <format>` | Export format: `json` (default) or `csv`. |
| `--out-file <path>` | Write to a file instead of stdout. |

`summary` takes only the positional `run-id`.

```bash
agnt5 reports summary run_9f2a
agnt5 reports failures run_9f2a --page-size 50
agnt5 reports wait run_9f2a --timeout 15m
agnt5 reports export run_9f2a --artifact-format csv --out-file results.csv
```

<Callout type="tip">
`agnt5 reports ci`/`agnt5 reports wait` return a non-zero exit code when the CI gate fails (default `--fail-on-gate=true`) — wire either into a pipeline step to block merges on a regression.
</Callout>

## `agnt5 scores`

```bash
agnt5 scores <command> [options]
```

| Command | Purpose |
| --- | --- |
| `list` | Query individual scorer outputs with filters. |
| `evidence <score-id>` | Fetch a score's input, output, and evidence artifacts. |

#### `scores list`

Filter by any combination of the following:

| Flag | Description |
| --- | --- |
| `--subject-type <type>` | Subject kind, e.g. `experiment_run_item`, `span`, `session`, `fleet_run`. |
| `--subject-id <id>` | Specific subject ID. |
| `--scorer-id <id>` | Filter by scorer. |
| `--scorer-version-id <id>` | Filter by scorer version. |
| `--run-id <id>` | Filter by experiment run. |
| `--run-item-id <id>` | Filter by run item. |
| `--session-id <id>` | Filter by session. |
| `--root-run-id <id>` | Filter by root run. |
| `--journal-id <id>` | Filter by journal. |
| `--span-id <id>` | Filter by span. |
| `--component-type <type>` | Filter by component type. |
| `--component-name <name>` | Filter by component name. |
| `--since <time>` | Lower time bound (RFC3339 or Unix timestamp). |
| `--until <time>` | Upper time bound (RFC3339 or Unix timestamp). |
| `--page <n>` | Page number. Default: `1`. |
| `--page-size <n>` | Results per page. Default: `50`. |

#### `scores evidence <score-id>`

| Flag | Description |
| --- | --- |
| `--include <artifacts>` | Comma-separated artifacts to include: `scorer_input`, `scorer_output`, `evidence`. |

```bash
agnt5 scores list --run-id run_9f2a --page-size 100
agnt5 scores evidence score_4b21 --include scorer_input,scorer_output,evidence
```

## Next steps

- [Run commands](/docs/cli/run.md): produce the runs that `datasets add-run` imports.
- [Deploy commands](/docs/cli/deploy.md): experiments can target a deployment instead of a local component.
- [Automation & agents](/docs/cli/automation.md): script the eval loop from CI with `--output json`.
- [Inspect & logs](/docs/cli/inspect.md): trace an individual run behind an eval result.