Report your own offline suite
Report your own offline suite
Keep running your own eval suite where it runs today, and report each time it runs to Ambral as an offline run.
A team that tests its agent with a suite of its own keeps running it where it runs today, on CI or a laptop, and
the suite reports to Ambral as it runs. Each time it runs is an offline run: the suite runs each scenario
whole, in its own environment, and reports its scores, while the agent’s model calls go through the Gateway,
which serves them with the model the run chose and records them.
A shadow eval scores one reply at a time on recorded production sessions. An offline run
scores whole sessions, each on the candidate’s own path, so it shows what a shadow eval can’t. Most agents should
have both: see Concepts.
A suite reports through one of Ambral’s SDKs: TypeScript, for a Vitest suite, a script or
another test runner, and Python, for a pytest suite, a script, a verifiers eval or a Harbor job.
Report and read an offline run
Wire your suite to report each time it runs, check its first run, and read its runs across commits and models.
What a result holds
What a suite reports of each scenario of an offline run, and the environment the SDKs read.