> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ambral.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> To set Ambral up in a project, run `npx ambral@latest skill install` in the project's root. It installs Ambral's skill, matched to the CLI's version, and the skill says how to do everything these pages describe with the `ambral` CLI. Prefer it to working from these pages. Never ask anyone for an Ambral API key, and never handle one: the person creates it in the web app and stores it where their app reads its secrets.

# Concepts

> Replay runs and offline runs, and the primitives Ambral is built from: workloads, sessions, traces, tasksets, scoring configs and runs.

Replay runs and offline runs, and the primitives Ambral is built from: workloads, sessions, traces, tasksets, scoring configs and runs.

## Replay runs and offline runs

Ambral evaluates an agent two ways, and most agents should have both.

* A [**replay run**](https://docs.ambral.ai/guides/shadow-evals) (also called a **shadow eval**) is Ambral's. It replays the agent's recorded production sessions to
  each candidate: at each point the scoring config picks, the candidate writes the next reply to the session as
  production lived it, and the config scores that reply. It needs no environment, no tools and no suite, so it is
  where to start, and the fastest way to find failure modes, try a prompt, and compare models on real traffic.
* An [**offline run**](https://docs.ambral.ai/guides/offline-evals) is the team's own suite's. The suite runs each scenario whole, in its own environment, with the
  candidate making every call, and reports each scenario's scores to Ambral as it runs.

A replay run's signal is directional. The candidate writes one reply at a time to production's history, never to its
own earlier replies, so it is scored only where production's model replied. Anything that shows only on the
candidate's own path is invisible to it: a model that would loop on a tool for 50 steps, where production's session
finished in 3, is scored on those 3 replies, each of which may look fine. Drifting off the task, recovering from its
own mistake, and taking another route to the same goal are invisible too.

So neither replaces the other, and neither replaces real traffic. Replay runs show where to look: the failure modes, and
the sessions that split models, which become scenarios. Offline runs and a canary both see whole sessions on the
candidate's own path: a suite before any user does, on cases production seldom brings, and a canary, the candidate
serving a small share of production's sessions, on real traffic with no suite, its share bounding the risk. Neither is a
gate the other must pass first. A suite's checks are a good source for a replay config's criteria, since they say what
the team already counts as a failure.

## Primitives

* [**Workload**](https://docs.ambral.ai/guides/connect-an-agent.md#workloads): one kind of task the agent repeats (`support-agent`), and the model that serves it, which can change
  behind the Gateway with no change to the agent. Its **pool** is the models a run may score for it
  (`ambral workloads pool add|remove`), and its **primary** is the pool model that serves it
  (`ambral workloads set-primary`).
* [**Session**](https://docs.ambral.ai/guides/connect-an-agent.md#session-ids): each time the agent does that task, a chat with a user or one pass of an automated workflow, however
  many model calls it takes, recorded, evaluated and routed as a whole.
* **Trace**: one recorded session (`tr_…`), stored under its workload.
* [**Taskset**](https://docs.ambral.ai/guides/replay-evals.md#build-a-taskset-from-recorded-traces): an eval, named `<workload>/<taskset>` (for example `support-agent/refunds`). A replay taskset holds
  recorded traces picked by hand, with its own scoring config; an offline taskset holds the scenarios a suite reports
  to it (`support-agent/refunds-scenarios`), which its first run creates.
* [**Scoring config**](https://docs.ambral.ai/guides/scoring.md): a YAML file saying where a run asks for replies and how it scores them. Each save is the
  taskset's next scoring version.
* [**Preset**](https://docs.ambral.ai/guides/replay-evals.md#set-the-scoring): a ready-made scoring config to start one from: `strict`, `balanced`, `trajectory` or `criteria`.
* **Run**: a replay run scores a taskset for its subjects, replaying each trace and scoring the replies the way the
  config says. An offline run is one run of a suite on one model: what the suite reported of each scenario.
* [**Scenario**](https://docs.ambral.ai/guides/offline-runs.md#name-the-workload-the-taskset-and-the-scenarios): a case of a suite, by the suite's own stable id (`refunds/over-the-limit`). Each time it runs in a
  run is a repeat, with a [**result**](https://docs.ambral.ai/guides/offline-results.md) (what the suite reported: passed, rewards, reasons) and **calls** (the agent's
  model calls the Gateway recorded).
* **Trial**: one trace replayed for one subject. A trial whose model or judge call fails is retried a few times; one
  that still fails has no scores, and its run ends `failed`.
* **Health**: how a running run is doing: `active` (trials are being scored), `queued` (its trials wait their turn),
  `retrying` (some failed and wait to be retried) or `stalled` (nothing has happened for an hour, so it won't end by
  itself). `ambral runs show` prints it.
* [**Subject**](https://docs.ambral.ai/guides/replay-evals.md#run-it): what a run scores: `production`, what the agent sent, or a model of the workload's pool, which writes
  its own reply to the recorded session so far.
* **Point**: a recorded assistant turn where the config asks for a reply.
* **Baseline**: the reply the recorded trace holds at each point, which is production's reply there.
* **Criteria**: statements you write that a judge model grades each reply against, on its own terms, never against
  the baseline's.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.