Skip to main content
Report a Vitest suite, a script or another test runner to Ambral as an offline run. The Gateway’s base URL is the agent’s own setting: connection.baseUrl from ambral workloads get <workload> --json, under whatever name the agent reads it (AMBRAL_GATEWAY_URL below).

Vitest

npm install --save-dev @ambral/evals, which needs Node 22 or later and Vitest 5.0.4 or later. The suite gets a Vitest config of its own, apart from the unit tests’, which takes only its eval files and names the suite:
Its files use the SDK’s test:
  • Run it with vitest run --config vitest.evals.config.ts. Plain vitest watches, and each rerun opens a run of its own. Never spread ambral() into the unit tests’ config: its globalSetup and provide replace theirs, and every unit run would open a run and need AMBRAL_API_KEY.
  • ./agent.ts stands for the suite’s own agent. Its model client sends scenario.headers on each call: as its default headers (new OpenAI({ baseURL: process.env.AMBRAL_GATEWAY_URL, defaultHeaders: scenario.headers })), or on each request.
  • Each test of the SDK’s test is a scenario. Its id is the test’s file and full name (refunds.eval.ts > over the limit), so renaming a test or its file starts a new scenario.
  • A test that passes reports passed, and one that fails reports its errors. scenario.score(name, score, reason) adds a reward (vitest-evals’ scores are read too), and scenario.task(data) sets the scenario’s task.
  • An agent that names its own sessions takes scenario.session as its session id.
  • ambral({ … }) also takes model, labels, key and source. A config of several projects spreads one into each project’s test, and each opens a run of its own suite.

A TypeScript script, or another test runner

  • scenario.report(result) resolves once Ambral stored it, and reports go together, about once a second: collect them and wait for them together, not one after another.
  • run.close() sends what is left, prints the run’s id and the command that reads it, and rejects naming each scenario this process ran that has no result, a refused report’s reason included. Close in a finally: a run nobody closed stays open.
  • run.scenario(id, { session }) takes the agent’s own session id.

A model client built once

A client built once, at module scope, can’t take each scenario’s headers as its defaults. Build it with the SDK’s transport instead, which adds the current scenario’s headers to each call, and refuses a call made outside a scenario rather than send it as production’s. In TypeScript, the transport is fetch from @ambral/evals: new OpenAI({ baseURL: process.env.AMBRAL_GATEWAY_URL, fetch }). In a test of @ambral/evals/vitest, the test’s scenario is the current one. A script runs each scenario’s agent within scenario.run(() => …).