> For the complete documentation index, see [llms.txt](/llms.txt).
> A full single-fetch corpus is available at [llms-full.txt](/llms-full.txt).
---
title: Scorers
description: Score component outputs with built-in deterministic checks, trace assertions, LLM-as-judge presets, or your own custom scorer code.
last_verified: 2026-10-08
---



A **scorer** decides whether a component's output meets the target behavior. Each scorer returns a score between 0.0 and 1.0, a pass/fail verdict, and an optional explanation. [Experiments](/docs/improve/experiments.md) attach one or more scorers and run them against every [dataset](/docs/improve/datasets.md) item.

---

## Scorer classes

AGNT5 has exactly three scorer classes:

| Class | Examples | Needs deployment? |
|---|---|---|
| **Built-in deterministic** | `exact_match`, `json_schema`, `tool_called` | No — runs as AGNT5-owned logic |
| **Built-in LLM-as-judge** | `llm_judge`, `correctness`, `faithfulness` | No — AGNT5-owned, configurable model and rubric |
| **Custom** | Your `@scorer` functions | Yes — registered and deployed with your worker |

Built-ins work out of the box: select them in Studio or pass them to `agnt5 experiments create --builtin-scorer <name>`. Only custom scorers require your own code.

Scorers also differ by what they evaluate:

- **Output scorers** compare a single item's output against its input and expected output.
- **Trace scorers** assert on execution behavior — which tools were called, how many LLM calls were made, how long the run took. They need trace events, which dataset items carry when [imported from production runs](/docs/improve/datasets.md#from-production-runs).

---

## Built-in deterministic scorers

Output scorers:

| Scorer | Checks |
|---|---|
| `exact_match` | Output equals the expected output exactly |
| `contains` | Output contains a substring |
| `regex_match` | Output matches a regular expression |
| `json_valid` | Output is well-formed JSON |
| `json_schema` | Output validates against a JSON Schema |
| `numeric_range` | Numeric output falls within a range |
| `levenshtein` | Output is similar to expected by edit distance |
| `structured_assertions` | Configured assertions over input, output, and expected JSON |

Trace scorers:

| Scorer | Checks |
|---|---|
| `tool_called` / `tool_not_called` | A named tool was (or was not) called |
| `tool_sequence` / `tool_sequence_in_order` | Tools were called in the configured order |
| `tool_sequence_any_order` | Configured tools all appear, in any order |
| `tool_trajectory` | Tools match a selected trajectory pattern (`exact`, `in_order`, or `any_order`) |
| `max_tool_calls` / `max_llm_calls` | Total tool or LLM calls stay under a limit |
| `max_tokens` | Total LLM tokens stay under a budget |
| `duration_under` | Session duration stays under a limit |
| `no_errors` | The execution produced no errors |
| `state_equals` | A named state snapshot equals an expected value |

Pass a bare name for default behavior, or a JSON object with a `config` for the scorers that need one:

| Scorer | Config |
|---|---|
| `exact_match`, `json_valid`, `levenshtein`, `no_errors` | None |
| `contains`, `regex_match` | `pattern` (required) |
| `json_schema` | `schema`, a JSON Schema object (required) |
| `numeric_range` | `min` and `max` |
| `structured_assertions` | `assertions` (required) |
| `tool_called`, `tool_not_called` | `tool`, the tool name (required) |
| `tool_sequence`, `tool_sequence_in_order`, `tool_sequence_any_order` | `tools`, an array of tool names (required) |
| `tool_trajectory` | `tools` (required) and `pattern`: `exact`, `in_order`, or `any_order` |
| `max_tool_calls`, `max_llm_calls`, `max_tokens` | `max` (required) |
| `duration_under` | `max_ms` (required) |
| `state_equals` | `name` and `expected` (required) |
| `llm_judge` | `criteria` (or `prompt_template`) and `model` (required) |
| `faithfulness` | `context_fields`, such as `["input.message"]` (required) |

```bash
agnt5 experiments create \
  --name support-agent-quality \
  --dataset-id <dataset-id> \
  --dataset-version-id <dataset-version-id> \
  --deployment-id <deployment-id> \
  --component-name support_agent \
  --component-type agent \
  --builtin-scorer json_valid \
  --builtin-scorer '{"name":"tool_called","config":{"tool":"search_orders"}}' \
  --builtin-scorer '{"name":"max_llm_calls","config":{"max":5}}'
```

### Structured assertions

`structured_assertions` runs 1–64 boolean expressions per item. The score is the fraction that pass; `score_threshold` (default `1`) sets the pass mark.

```bash
--builtin-scorer '{"name":"structured_assertions","config":{"assertions":[
  {"name":"is_list","expr":"is_array(output_json)"},
  {"name":"length_matches","expr":"size(output_json) == expected.expected_length"}
]}}'
```

Expressions support only:

| Kind | Supported |
|---|---|
| Roots | `input`, `output`, `expected`, and dot-separated fields such as `output.answer`. `input_json`, `output_json`, and `expected_json` parse a JSON-encoded string first. |
| Literals | JSON strings, numbers, `true`, `false`, `null` |
| Operators | `==`, `!=`, `<`, `<=`, `>`, `>=`, `&&`, `\|\|`, `!`, parentheses. Ordering compares numbers only; equality is structural. |
| Functions | `size(x)` (array, object, or string length), `unique(array)`, `all(array, predicate)`, `any(array, predicate)` |
| Predicates | `is_array`, `is_object`, `is_string`, `is_number`, `is_boolean`, `is_null` |

There are no string functions (`contains`, `starts_with`, regular expressions), method calls such as `output.contains(...)`, indexing, or arithmetic. For text checks, use the `contains` or `regex_match` scorers (fixed pattern) or a [custom scorer](#custom-scorers) when the check depends on each item's expected value.

AGNT5 parses every expression when you create or update the experiment. An invalid one is rejected with the assertion name, the error, and its character position:

```text
builtin scorer "structured_assertions" assertions[0] ("answer_in_output"): unknown function "string" at position 0 in "string(output).contains(expected.answer)"; supported functions: size, unique, all, any, is_array, is_object, is_string, is_number, is_boolean, is_null
```

---

## Built-in LLM-as-judge scorers

LLM-as-judge scorers use a language model to grade outputs against criteria. All are AGNT5-owned — no registration or deployment needed — and accept overridable judge settings: model, provider, prompt, and rubric.

| Scorer | What it grades |
|---|---|
| `llm_judge` | Generic judge — you supply the criteria and rubric |
| `correctness` | Output correctly answers the input and matches expected output |
| `faithfulness` | Output stays faithful to configured context fields; penalizes hallucination |

Pass them to an experiment the same way as deterministic scorers:

```bash
agnt5 experiments create \
  --name my-experiment \
  --dataset-id <dataset-id> \
  --dataset-version-id <dataset-version-id> \
  --deployment-id <deployment-id> \
  --component-name my_agent \
  --component-type agent \
  --builtin-scorer correctness \
  --builtin-scorer '{"name":"llm_judge","config":{"criteria":"Is the response concise and actionable?","model":"gpt-6-luna"}}'
```

Give `model` as the provider's model name, for example `gpt-6-luna`, and set `provider` when it is not `openai`. Use one `llm_judge` per experiment; to grade several criteria, put them in one `criteria` string or use the preset scorers such as `correctness` and `faithfulness`.

> **Note:** Judge scorers call an LLM provider, so the project needs the provider credential configured (for example `OPENAI_API_KEY` as a [project secret](/docs/run/deploying.md#secrets)).

### SDK evaluator presets

All three SDKs export typed presets for common LLM judge patterns. Use these with `client.eval()` and `client.batch_eval()` (`client.Eval` / `client.BatchEval` in Go) instead of raw scorer strings:

| Preset class | Grades |
|---|---|
| `Correctness` | Output correctly answers the input and matches expected |
| `Faithfulness` | Output is faithful to provided context; penalizes unsupported claims |
| `Helpfulness` | Output is useful, complete, and actionable for the user's task |
| `Coherence` | Output is logically organized and internally consistent |
| `Conciseness` | Output is concise while preserving all needed information |
| `ResponseRelevance` | Output directly addresses the input without off-topic content |
| `InstructionFollowing` | Output follows all explicit and implied instructions |
| `GoalSuccess` | Session achieved the user's goal (uses journal events and session state) |
| `Refusal` | Refusals are appropriate, well-explained, and offer safe alternatives |
| `Harmfulness` | Output avoids instructions or claims that could enable harm |
| `Stereotyping` | Output avoids stereotypes and biased generalizations |



**Python:**

```python
from agnt5.eval import Correctness, Helpfulness, Faithfulness

scorers = [
    Correctness(),
    Helpfulness(model="openai/gpt-6-luna"),
    Faithfulness(context_fields=["retrieved_chunks"]),
]
```

All presets accept an optional `model`, `threshold` (default `0.7`), and `include_input`. See [Batch eval](/docs/improve/batch-eval.md) for full SDK usage.





**TypeScript:**

```typescript
import { Correctness, Helpfulness, Faithfulness } from '@agnt5/sdk';

const scorers = [
  new Correctness(),
  new Helpfulness({ model: 'openai/gpt-6-luna' }),
  new Faithfulness({ contextFields: ['retrieved_chunks'] }),
];
```

All presets accept an optional `model`, `threshold` (default `0.7`), and `includeInput`. See [Batch eval](/docs/improve/batch-eval.md) for full SDK usage.





**Go:**

```go
helpfulness := agnt5.Helpfulness{}
helpfulness.Model = "openai/gpt-6-luna"

faithfulness := agnt5.Faithfulness{}
faithfulness.ContextFields = []string{"retrieved_chunks"}

scorers := agnt5.NormalizeEvalScorers(agnt5.Correctness{}, helpfulness, faithfulness)
```

Each preset is a struct that embeds `agnt5.EvaluatorPresetConfig`, so `Model`, `Threshold` (`*float64`, default `0.7`), `IncludeInput` (`*bool`), and `ContextFields` are set as promoted fields. Every preset has a `ToEvalScorerSpec()` method that yields the `agnt5.EvalScorerSpec` the client expects; `agnt5.NormalizeEvalScorers` converts a mix of names, presets, and `agnt5.NewLLMJudge` values in one call. See [Batch eval](/docs/improve/batch-eval.md) for full SDK usage.



---

## Custom scorers

When built-ins can't express your check, write a **custom scorer** — user code that receives the eval context and returns a result. Custom scorers are components: they register with your worker and deploy with your code.



**Python:**

```python
from agnt5 import scorer, ScorerContext, ScorerRequest, ScorerResult

@scorer(name="cites_order_id", description="Reply must cite the order ID from the input")
async def cites_order_id(ctx: ScorerContext, request: ScorerRequest) -> ScorerResult:
    order_id = request.input.get("order_id", "")
    cited = order_id in str(request.output)
    return ScorerResult(
        score=1.0 if cited else 0.0,
        passed=cited,
        explanation=f"Order ID {order_id} {'found' if cited else 'missing'} in reply",
    )
```

The `ScorerRequest` carries `input`, `output`, `expected`, `config`, and `trace` (trace events for trace-level scorers, declared with `scope="trace"`). Import `scorer` from `agnt5`, the entry point that registers the scorer with your worker.





**TypeScript:**

```typescript
import { scorer, ScorerResult } from "@agnt5/sdk";

const citesOrderId = scorer("cites_order_id", "Reply must cite the order ID from the input")(
  async (ctx, request) => {
    const orderId = (request.input as { order_id?: string }).order_id ?? "";
    const cited = String(request.output).includes(orderId);
    return new ScorerResult({ score: cited ? 1 : 0, passed: cited });
  },
);
```





**Go:**

```go
import "github.com/agnt5dev/sdk-go/agnt5"

citesOrderID := agnt5.ScorerConfig{
    Name:        "cites_order_id",
    Description: "Reply must cite the order ID from the input",
    Handler: func(ctx context.Context, req agnt5.ScorerRequest) (agnt5.ScorerResult, error) {
        input, _ := req.Input.(map[string]any)
        orderID, _ := input["order_id"].(string)
        cited := strings.Contains(fmt.Sprint(req.Output), orderID)
        score := 0.0
        if cited {
            score = 1.0
        }
        return agnt5.ScorerResult{Score: score, Passed: cited}, nil
    },
}

err := agnt5.RegisterScorer(worker, citesOrderID)
```

`ScorerRequest.Input`, `Output`, and `Expected` are typed `any`, so assert them to the JSON shape you expect (`map[string]any`, `string`, ...) before indexing. The Go SDK ships two built-in deterministic scorer constructors you can also register this way: `agnt5.ExactMatchScorer()` and `agnt5.ContainsScorer()` — a much smaller built-in set than the CLI's full list (see [Built-in deterministic scorers](#built-in-deterministic-scorers) above), so most checks still need a custom `ScorerConfig`.



Custom scorers register with the worker like any other component: pass them in the worker's `scorers` list. After [deploying](/docs/run/deploying.md), create a scorer record that points at the deployed scorer, then attach the record to an experiment by its ID:

```bash
curl -X POST "https://api.agnt5.com/api/v1/projects/<project-id>/scorers" \
  -H "X-API-KEY: <api-key>" \
  -H "Content-Type: application/json" \
  -d '{"name": "cites_order_id", "type": "deployed", "deployment_id": "<deployment-id>", "component_name": "cites_order_id"}'

agnt5 experiments create ... --scorer-id <scorer-record-id>
```

The API call takes a personal API key (see [Authentication](/docs/api-reference/authentication.md)) and returns the record's `id`. See [Create a scorer](/docs/api-reference/scorers/create-a-scorer.md).

---

## Trace assertions in the SDK

For glassbox testing — asserting on execution behavior rather than output content — the SDK provides `TraceAssertion` and `trace_scorer()` / `traceScorer()` / `agnt5.TraceScorer()`. These are used inside custom scorers or directly in [batch eval](/docs/improve/batch-eval.md) code; they do not map to CLI scorer names.



**Python:**

```python
from agnt5 import scorer, ScorerContext, ScorerRequest, ScorerResult
from agnt5.eval import TraceAssertion, trace_scorer

@scorer(name="efficiency_check", scope="trace")
async def efficiency_check(ctx: ScorerContext, request: ScorerRequest) -> ScorerResult:
    assertions = [
        TraceAssertion.max_tokens(2000),
        TraceAssertion.max_lm_calls(4),
        TraceAssertion.no_errors(),
        TraceAssertion.duration_under(15000),
    ]
    result = trace_scorer(request.trace or [], assertions)
    return ScorerResult(
        score=result.score,
        passed=result.passed,
        explanation=result.explanation,
    )
```





**TypeScript:**

```typescript
import { scorer, TraceAssertion, traceScorer } from "@agnt5/sdk";

const efficiencyCheck = scorer("efficiency_check", "Checks token use, LLM calls, and duration", "trace")(
  async (ctx, request) => {
    const result = traceScorer(request.trace ?? [], [
      TraceAssertion.maxTokens(2000),
      TraceAssertion.maxLmCalls(4),
      TraceAssertion.noErrors(),
      TraceAssertion.durationUnder(15000),
    ]);
    return { score: result.score, passed: result.passed, explanation: result.explanation };
  },
);
```





**Go:**

```go
import (
    "context"
    "time"

    "github.com/agnt5dev/sdk-go/agnt5"
)

efficiencyCheck := agnt5.ScorerConfig{
    Name:        "efficiency_check",
    Description: "Checks token use, LLM calls, and duration",
    Scope:       agnt5.ScorerScopeTrace,
    Handler: func(_ context.Context, req agnt5.ScorerRequest) (agnt5.ScorerResult, error) {
        return agnt5.TraceScorer(req.Trace, []agnt5.TraceAssertion{
            agnt5.MaxTokens(2000),
            agnt5.MaxLMCalls(4),
            agnt5.NoErrors(),
            agnt5.DurationUnder(15 * time.Second),
        }), nil
    },
}

err := agnt5.RegisterScorer(worker, efficiencyCheck)
```

The assertions are package-level constructors (`agnt5.MaxTokens(n)`, not methods on `TraceAssertion`), `req.Trace` carries the item's `[]agnt5.TraceEvent`, and `agnt5.TraceScorer` returns an `agnt5.ScorerResult` you can hand back directly.



Available assertions:

| Python / TypeScript | Go | Asserts |
|---|---|---|
| `max_tokens(n)` / `maxTokens(n)` | `agnt5.MaxTokens(n)` | Total LLM tokens ≤ n |
| `max_lm_calls(n)` / `maxLmCalls(n)` | `agnt5.MaxLMCalls(n)` | LLM call count ≤ n |
| `no_errors()` / `noErrors()` | `agnt5.NoErrors()` | No error events in the trace |
| `duration_under(ms)` / `durationUnder(ms)` | `agnt5.DurationUnder(d)`, a `time.Duration` | Execution duration < ms |
| `event_sequence([...])` / `eventSequence([...])` | `agnt5.EventSequence([]string{...})` | Named events appear in order |
| `step_memoized(name)` / `stepMemoized(name)` | `agnt5.StepMemoized(name)` | Step was served from cache |
| `event_count(type, min)` / `eventCount(type, min)` | `agnt5.EventCount(type, min)` | Event type appeared ≥ min times |

`trace_scorer()` / `traceScorer()` / `agnt5.TraceScorer()` returns a score equal to the proportion of assertions that passed, plus a combined explanation.

---

## Inspect scores

Every scorer execution produces a **score** record with evidence — the inputs the scorer saw and why it decided what it decided.

```bash
# All scores for an experiment run
agnt5 scores list --run-id <run-id>

# Scores for a specific run item
agnt5 scores list --run-id <run-id> --run-item-id <item-id>

# Scores produced by a specific scorer
agnt5 scores list --scorer-id <scorer-id>

# Scores created since a timestamp (RFC3339 or Unix seconds)
agnt5 scores list --since 2026-10-08T09:00:00Z

# Live production scores (for online evals)
agnt5 scores list --root-run-id <root-run-id>
agnt5 scores list --session-id <session-id>

# Show evidence for one score — scorer input, output, and reasoning
agnt5 scores evidence <score-id> --include scorer_input,scorer_output,evidence
```

See [`scores list`](/docs/cli/evals.md#scores-list) for every filter.

In Studio, open **Evaluate** -> **Experiments**, select a run, and click into any item to see its per-scorer results and evidence.


**Scorer classes**: built-in deterministic (SDK-core owned, no registration), built-in LLM-as-judge (`llm_judge`, `correctness`, `faithfulness`; configurable model/prompt/rubric), custom (user components, require worker registration + deployment).
**Built-in deterministic names**: `exact_match`, `contains`, `regex_match`, `json_valid`, `json_schema`, `numeric_range`, `levenshtein`, `structured_assertions`, `tool_called`, `tool_not_called`, `tool_sequence`, `tool_sequence_in_order`, `tool_sequence_exact`, `tool_sequence_any_order`, `tool_trajectory`, `tool_params_match`, `max_tool_calls`, `max_llm_calls`, `max_tokens`, `duration_under`, `no_errors`, `state_equals`.
**SDK evaluator presets** (wrap `llm_judge` or named built-ins): `Correctness`, `Faithfulness`, `Helpfulness`, `Coherence`, `Conciseness`, `ResponseRelevance`, `InstructionFollowing`, `GoalSuccess`, `Refusal`, `Harmfulness`, `Stereotyping`. Go: same names as structs embedding `agnt5.EvaluatorPresetConfig`, each with `ToEvalScorerSpec()`; `agnt5.NormalizeEvalScorers(...)` mixes names, presets, and `agnt5.NewLLMJudge(LLMJudgeConfig{...})`.
**TraceAssertion** (glassbox, used inside custom scorers with `scope="trace"`): `max_tokens`, `max_lm_calls`, `no_errors`, `duration_under`, `event_sequence`, `step_memoized`, `event_count`; composed with `trace_scorer()`. Go: `agnt5.MaxTokens`, `MaxLMCalls`, `NoErrors`, `DurationUnder(time.Duration)`, `EventSequence`, `StepMemoized`, `EventCount`, composed with `agnt5.TraceScorer(trace, assertions) ScorerResult` inside a `ScorerConfig` with `Scope: agnt5.ScorerScopeTrace`.
**Result shape**: `{score: 0.0–1.0, passed: bool, label?: string, explanation?: string, metadata?: object}`.
**Code primitives**: `@scorer(name, description, scope)` decorator + `EvalContext` -> `ScorerResult` (Python); `scorer(name, description, scope)(handler)` (TypeScript); `agnt5.RegisterScorer(worker, agnt5.ScorerConfig{Name, Description, Scope, Handler})` with `Handler func(context.Context, ScorerRequest) (ScorerResult, error)`, where `ScorerRequest.Input`/`Output`/`Expected` are `any` and `Trace` is `[]TraceEvent` (Go).
**Errors**: built-ins fail with typed scorer errors (`input_error`, `config_error`, `provider_error`, `auth_error`, `artifact_error`, `timeout_error`); only custom scorers can return `scorer_not_found`.


## Next steps

* [Experiments](/docs/improve/experiments.md): attach scorers to an experiment and run them against a dataset version.
* [Batch eval](/docs/improve/batch-eval.md): run scorers from code using `client.eval()` or `client.batch_eval()`.
* [Datasets](/docs/improve/datasets.md): curate the items your scorers grade, including trace events for trace scorers.
* [Agents](/docs/build/agents.md): structure agent tool use so trace assertions have meaningful events to check.