Your coding agent can do everything on this page for you: see the Quickstart. This page says
what it does, and why.
What an offline run is
A team that tests its agent with a suite of its own (Vitest, pytest, verifiers, Harbor, a script) keeps running it where it runs today, on CI or a laptop, and it reports to Ambral as it runs. The suite runs each scenario and scores it. The agent’s model calls go through the Gateway, which serves them with the model the run chose and records them. Each time the suite runs is an offline run. An offline run holds two things for each run of a scenario:- its result: what the suite reported, which is whether it passed, its rewards and the reasons for them;
- its calls: the agent’s model calls the Gateway recorded, as a trace.
Before you wire it
- The suite’s agent has to be on the Gateway. An offline run records only the calls that go through it. A suite whose agent calls a provider with a key of its own, or answers from a mocked model or recorded cassettes, can’t report one as it is: connect its agent first, as you would for production.
- The suite needs a key. It reports with an Ambral key, the same kind its agent calls the Gateway with, read
from
AMBRAL_API_KEY. Set it where the suite runs: your shell or a.envfile that git ignores on a laptop, and the CI’s secrets on CI. The SDK’s settings lists the rest of what it reads from the environment. - Note each client that calls a model in a scenario. Besides the agent’s, a suite often has a judge, a simulated user, or a second agent the first one calls. Only some of them get the run’s headers.
Name the workload, the taskset and the scenarios
- The workload is the agent’s, the one its production calls already name. Its pool, which
ambral workloads getlists, is the models a run may choose. Add one withambral workloads pool add. - The taskset is the suite’s name in Ambral,
<workload>/<taskset>. Its first run creates it. Lead with the kind of case and end with-scenarios, so the suite sorts beside the replay tasksets of the same kind:support-agent/refundsandsupport-agent/refunds-scenarios. Every run of the suite, on every model and every commit, reports to the same taskset, which is what lets its runs be lined up. A taskset a typo created can’t be deleted, so check the name before the first run. - A scenario’s id is what lines a scenario up across runs, so keep it stable: the same case keeps its id from
run to run and commit to commit. Write it as a path, most general first, such as
refunds/over-the-limit, since the commands that read runs roll scenarios up by the segments of their ids. Never put a run’s index, a timestamp or a random id in it: every run would then be a suite of new scenarios, and no scenario would have a history. An id is at most 512 characters. - A check’s name, a reward’s or a metric’s, is how checks are lined up across runs. Name a check by the rule
it tests, such as
policy_followedorsolved.
Pick a harness
- TypeScript:
@ambral/evals: a Vitest suite, a script or another test runner. - Python:
ambral-evals: a pytest suite, a script, a verifiers eval or a Harbor job.
Which clients get the headers
Every model call of the system under test carries the scenario’s headers, and nothing else does.- The agent’s client: always. Its calls are served the run’s model and recorded as the scenario’s session.
The agent keeps sending
ambral/autoas its model: the headers say which workload, run and model serve it. - A judge’s, or a simulated user’s: never. Give them a client of their own, without the headers. A call that carries them is served the run’s model, so the judge or the user changes with the model being tested, and its messages are recorded in the agent’s conversation.
- Another workload the agent calls, such as a router or a sub-agent: give its client the headers too. A call that names another workload is served that workload’s primary model, as in production, and recorded under the run beside the scenario’s session.
What a result holds
See What a result holds.Repeats, labels and the model
- Repeats. A model answers a little differently each time, so a scenario that ran once can pass or fail by
luck. Run each scenario 3 times or more where you can afford it. Each run of a scenario in one run is a repeat,
and the commands that read runs count the repeats passed, as
2/3. In Vitest, setrepeatson the test, notretry, which runs a test again only after it failed. In verifiers, setnum_rollouts, in Harbor--n-attempts, and in a script, loop. pytest has no repeats of its own: run the suite again under the sameAMBRAL_EVAL_KEY, which reopens the run and adds to its repeats. - Labels. Label each run with what you will look for it by: the commit and the branch it ran at, from the
CI’s own variables.
ambral tasksets resultsprints each run’scommit, andambral runs list --label branch=mainfinds a branch’s runs. - The model. A run is served one model: the one the code names, else
AMBRAL_EVAL_MODEL, else the workload’s primary when the run opens. It must be in the workload’s pool. To compare models, run the suite once per model, with no change to the code: a CI matrix that setsAMBRAL_EVAL_MODELper job, or one invocation per model on a laptop.
Cost and concurrency
A suite’s model calls are paid for with the org’s credits, at the same prices as production’s, and they draw on the same providers’ rate limits. With automatic top-up on, a suite’s calls can take the balance under its threshold and charge the saved card, as production’s calls do. An offline run has no budget. One run costs roughly its scenarios, times its repeats, times the agent’s calls in each: work that out before the first large run, and read what a run cost withambral runs show <run>.
Cap how many scenarios run at once, so a suite doesn’t slow production’s calls or hit a provider’s limits:
Vitest’s maxConcurrency and maxWorkers, pytest-xdist’s -n, verifiers’ max_concurrent, Harbor’s
--n-concurrent. A few dozen at once is plenty for most suites.
Where the suite runs
A run records where it ran, its source:ci when the CI environment variable is set, to anything but 0
or false, and local otherwise. A suite’s history leaves local runs out and counts them, so a developer trying
a change on a laptop doesn’t move the suite’s record. --source all reads them too.
- On GitHub Actions, GitLab CI, CircleCI or Buildkite there is nothing to set.
- On Jenkins and Azure Pipelines, which don’t set
CI, setAMBRAL_EVAL_SOURCE=ci, andAMBRAL_EVAL_KEYto the pipeline run’s id, so its jobs share one run. - On laptops only, or a scheduled machine, the history is empty until the runs of record say they are, with
AMBRAL_EVAL_SOURCE=ciwhere they run.
AMBRAL_EVAL_KEY. A process that crashes without closing leaves a shared run open:
ambral runs list --status open lists the open runs.
Check the first run
Run three scenarios before the whole suite, on a laptop or withAMBRAL_EVAL_SOURCE=local. A wiring mistake
shows in the first run, and costs less there.
1
Run them
The SDK’s last line names the run and the command that reads it:pytest shows the line only with
-rP. A report Ambral refused fails the suite with its reason: fix what it
says first.2
Wait half a minute, then read the run
- calls:
3 of 3 results have their agent's calls recorded. A result with no calls, 2 minutes after it was reported, means the agent’s model client didn’t send the run’s headers, or the agent names its own sessions and the suite didn’t pass the SDK its id. - results: every session that made calls with the run’s headers has a result. When some have none, which
ambral runs trace <run> --no-resultlists, the suite stopped before reporting them, or a client that isn’t the agent’s has the headers. - ids: from the suite’s second run on, how many scenario ids are new to the suite. All of them new means the ids aren’t stable.
3
Read one run of a scenario whole
tr_… that ambral runs trace <run> lists. Its conversation
should hold the agent’s calls only: no judge’s prompt, and no simulated user’s turn.4
Check the scores read as the suite meant them
ambral runs show <run> prints each reward’s mean and the reason given its lowest score, and
ambral runs verdicts <run> each result’s.Read a suite’s runs
A suite that reports its runs builds a history: a run per commit on CI, and a run per model when you compare them.ambral tasksets results lines them up by scenario:
Columns compare only on what they share, so check that a comparison is sound before you read it: that the
columns ran the same scenarios, at the same commit, with enough repeats to tell a failure from luck. Then read
the scenarios that split, not the averages, since a mean score hides which cases moved.