Skip to main content
Your coding agent can do everything on this page for you: see the Quickstart. This page says what it does, and why.
A replay eval is a taskset of your agent’s recorded sessions, with a scoring config that says where a run asks for replies and how it scores them. A replay run replays each session to each model you name: at each point the config picks, the model writes the next reply to the session as production lived it, and the config scores that reply. It needs no environment, no tools and no test suite, only recorded traces. A workload usually covers several kinds of request: a support agent handles refunds, shipping questions and account changes. Give each kind that matters its own taskset, named for it, such as support-agent/refunds, so each score says how a model does on that kind.

Build a taskset from recorded traces

List the workload’s sessions, and read the promising ones whole:
The list shows each trace’s first user message, its tools and its steps. --label env=prod keeps real traffic, when your agent labels its sessions, so a test harness’s sessions don’t end up in the taskset. Choose traces of one kind of request, with variety within it: different requests, different tools, short and long.
  • Pick the hard sessions, not only the common ones. A request just past a limit of your policy beside one just inside it, and the sessions built to mislead: an instruction planted in a message, a request from someone without the authority, a third contact about the same problem. That is where models differ. A taskset of easy sessions scores them all alike.
  • Pick sessions the recorded model got right, when the scoring will compare candidates with the baseline, the reply each trace recorded. Those replies are the answers every candidate is graded against, mistakes included, so read each reply before you pick its trace.
Then create the taskset from their ids:
A trace id takes the trace’s latest branch with a reply. tr_…:<node>, as ambral traces get names a branch, takes another. The branches are fixed when the taskset is created, so every run replays the same sessions. A taskset’s name is spelled as a workload’s is.

When the traffic is thin

Real traffic is mostly easy sessions, and a new workload has little of it. The sessions that tell models apart sit on the lines your agent’s policy draws: each limit, each rule that sends a case to a person, who may ask for what. Where the traces lack them, record them on purpose:
  1. Write inputs on both sides of each line, and the ones built to mislead, with each input’s right outcome.
  2. Run your agent on each as a scenario of an offline run, on a strong model of the workload’s pool. Its sessions are recorded as the run’s, apart from real traffic, and the run says which inputs the agent got right.
  3. Build tasksets from the run’s sessions as from any others: ambral runs trace <run> lists each one’s tr_….

When to make a new taskset

A taskset’s traces are fixed, and its scoring is versioned. When only how you grade changes, save the next scoring version on the same taskset. Make a new taskset, such as support-agent/refunds-v2, from traces recorded since, when what makes a good session has changed: a new policy, new tools, a different flow. Shipping a new system prompt is such a point too, since production should be the agent as it runs today.

Set the scoring

See How replay scoring works, and the examples, each a scoring config for one goal, starting with A support agent working tickets. A scoring config is a YAML file you keep in your repo, beside your agent’s code. Start it from a preset: ambral tasksets scoring preset lists them, each with when to use it, and prints one to start from. A preset with judges takes --judge, a model from ambral models list:
Every field of the file is explained in a comment above it. Edit it: remove the scorers the eval doesn’t need, and add the ones it does. Most evals need a few criteria of their own, statements you write that a judge model grades each reply against. Then check the file, and save it once it passes:
--check lists the mistakes it finds, each at its field, and saves nothing. Each save is the taskset’s next scoring version, so save once the file is right, not after each small change. To change the scoring later, edit the same file, then check and save it again. ambral tasksets scoring get writes the saved version back out, every default filled in.

Run it

A run scores a taskset for its subjects, which you name with --subjects:
  • production is what your agent sent: the replies the traces recorded. No model is asked for them, so it costs only the judges’ calls. Include it whenever you compare models: a score of 0.88 means little alone, and four points below production means something. Scorers that compare a reply with the baseline’s record nothing for it, since they would compare production with itself.
  • A model of the workload’s pool writes its own reply at each point. ambral workloads get lists the pool, and ambral workloads pool add adds a model to it.
  • A model under a prompt, <model>@<prompt>, is asked with that prompt in place of the system prompt each trace opens with. The prompt is a text file, or the pr_… id that ambral prompts put printed for one. It is scored in a column of its own, so production,<model>,<model>@pr_… reads the prompt’s effect beside the model’s own.
Start the run, and wait for it:
Each trace is replayed once per subject. A candidate’s replay calls its model at every point the config scores, then each of the config’s judges scores the reply. So a run’s time grows with its subjects, its traces and the points in each: three subjects over ten short traces can take more than ten minutes.
  • --wait stays until the run ends, then prints its summary. It gives up after 30 minutes, or after --timeout, and the run goes on. Without --wait, the command prints the run’s id at once: read the run with ambral runs show <run>, and find its id again with ambral runs list --taskset support-agent/refunds.
  • --budget caps what the run’s calls may cost, in USD. Once they cost that much the run is cancelled, and keeps what it scored before then. Calls already in flight still cost, so size it with room to spare. Nothing prices a run before it runs: to size a large one, run 2 or 3 of its traces first, and multiply the cost per finished trial that ambral runs show prints by the large run’s trials.
  • --scoring <file> scores this run alone under that file, without saving it to the taskset, so you can compare a draft with the saved version.
  • --label labels the run, such as --label experiment=shorter-prompt, so that ambral runs list --label experiment=shorter-prompt finds an experiment’s runs together.
While a run runs, ambral runs show prints its health: active, queued and retrying all mean it is on its way. stalled means nothing has happened for an hour, and it won’t end by itself. ambral runs cancel <run> stops a run in flight, and --subject stops one subject while the others finish. ambral runs resume <run> scores the trials a cancelled run left unfinished, under the same config. With --budget, it finishes a run its budget stopped, within a larger one that counts what the run has already spent. The trials it finished keep their results and aren’t paid for again. While a run is in flight, wait for it rather than starting it again, which pays for the same replies twice. Once it has ended, running it again is worth it to check that a result holds, since models and judges answer a little differently each time.

Read the results

Read a run from its summary down to single replies.
The summary prints each subject’s mean scores side by side, production’s column first:
  • Per trace. Each trace counts once, so compare models on this row.
  • Per scorer, by its name: each reward, metric, judge, criteria group and criterion. The count in brackets is how many scores are behind a mean: a criterion that is n/a at most points has few, so a 1.00 over 3 points says little.
  • Cost per trace is what a subject would cost to serve. A candidate’s is an estimate, and production’s is what production really cost.
Scores compare only within one scoring version, so compare runs whose header shows the same scoring vN. Then find where a subject fell short:
Each line is one scorer’s result on one reply: the trace, the point, the subject, the scorer, the judge’s verdict, the score and the judge’s reason. A scorer whose verdicts all agree with what you would say is working. One whose reasons don’t hold up is misfiring, and its config needs changing. Read one trace reply by reply:
At each point it shows what led up to it, the baseline’s reply, and each subject’s reply, with the model call that made it and each scorer’s verdict and reason. Read it before you trust a scorer or change it.

Change the scoring, and score again

When a scorer grades a reply differently from how you would, change the scoring file and rescore the run:
It judges the same replies again and calls only the judges. Save the file as the taskset’s next scoring version once its verdicts read the way you mean them.

Work on the failures

The traces where a scorer failed are the cases to work on. ambral runs verdicts with --check and --verdict prints them on its last lines, as Their traces: tr_…:<node>,…, which ambral tasksets create takes as they are. Run candidates on that focused taskset beside production, under the same scoring.

Promote a candidate

A candidate that beats production, on a result that holds when you run it again, may be worth serving. A replay run scores one reply at a time, so try the candidate on whole sessions first: with your own suite, as an offline run, or on part of production. For that, create a second workload that serves the candidate, send it a small share of your agent’s sessions, and score both workloads’ traces the same way. Then make the candidate the model the workload serves:
The Gateway serves it within about a minute, with no change to your agent. To roll back, run the same command with the model the workload served before.

Export a run

ambral runs export <run> > traces.jsonl writes every result as one line of JSON in verifiers’ episode format, to analyse the results or train on them outside Ambral. Export a run once it has ended: the file holds the whole run only when it had, and the command exited 0.