Skip to main content
A team that tests its agent with a suite of its own keeps running it where it runs today, on CI or a laptop, and the suite reports to Ambral as it runs. Each time it runs is an offline run: the suite runs each scenario whole, in its own environment, and reports its scores, while the agent’s model calls go through the Gateway, which serves them with the model the run chose and records them. A shadow eval scores one reply at a time on recorded production sessions. An offline run scores whole sessions, each on the candidate’s own path, so it shows what a shadow eval can’t. Most agents should have both: see Concepts. A suite reports through one of Ambral’s SDKs: TypeScript, for a Vitest suite, a script or another test runner, and Python, for a pytest suite, a script, a verifiers eval or a Harbor job.

Report and read an offline run

Wire your suite to report each time it runs, check its first run, and read its runs across commits and models.

What a result holds

What a suite reports of each scenario of an offline run, and the environment the SDKs read.