> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ambral.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> To set Ambral up in a project, run `npx ambral@latest skill install` in the project's root. It installs Ambral's skill, matched to the CLI's version, and the skill says how to do everything these pages describe with the `ambral` CLI. Prefer it to working from these pages. Never ask anyone for an Ambral API key, and never handle one: the person creates it in the web app and stores it where their app reads its secrets.

# TypeScript: @ambral/evals

> Report a Vitest suite, a script or another test runner to Ambral as an offline run.

Report a Vitest suite, a script or another test runner to Ambral as an offline run.

The Gateway's base URL is the agent's own setting: `connection.baseUrl` from `ambral workloads get <workload> --json`,
under whatever name the agent reads it (`AMBRAL_GATEWAY_URL` below).

## Vitest

`npm install --save-dev @ambral/evals`, which needs Node 22 or later and Vitest 5.0.4 or later. The suite gets a
Vitest config of its own, apart from the unit tests', which takes only its eval files and names the suite:

```ts theme={null}
// vitest.evals.config.ts
import { ambral } from "@ambral/evals/vitest/config"
import { defineConfig } from "vitest/config"

export default defineConfig({
  test: { include: ["**/*.eval.ts"], ...ambral({ workload: "support-agent", taskset: "refunds-scenarios" }) }
})
```

Its files use the SDK's `test`:

```ts theme={null}
// refunds.eval.ts
import { test } from "@ambral/evals/vitest"
import { expect } from "vitest"
import { reply } from "./agent.ts"

test("over the limit", async ({ scenario }) => {
  const message = "I want my $340 back for order 1044, it never came."
  scenario.task({ prompt: message, answer: "escalate" })
  const answer = await reply(message, scenario.headers)
  const escalated = answer.includes("escalate")
  scenario.score("solved", escalated ? 1 : 0, `Expected it to escalate; it replied: ${answer}`)
  expect(escalated).toBe(true)
})
```

* Run it with `vitest run --config vitest.evals.config.ts`. Plain `vitest` watches, and each rerun opens a run of its
  own. Never spread `ambral()` into the unit tests' config: its `globalSetup` and `provide` replace theirs, and every
  unit run would open a run and need `AMBRAL_API_KEY`.
* `./agent.ts` stands for the suite's own agent. Its model client sends `scenario.headers` on each call: as its
  default headers (`new OpenAI({ baseURL: process.env.AMBRAL_GATEWAY_URL, defaultHeaders: scenario.headers })`), or
  on each request.
* Each test of the SDK's `test` is a scenario. Its id is the test's file and full name
  (`refunds.eval.ts > over the limit`), so renaming a test or its file starts a new scenario.
* A test that passes reports `passed`, and one that fails reports its errors. `scenario.score(name, score, reason)`
  adds a reward (vitest-evals' scores are read too), and `scenario.task(data)` sets the scenario's task.
* An agent that names its own sessions takes `scenario.session` as its session id.
* `ambral({ … })` also takes `model`, `labels`, `key` and `source`. A config of several projects spreads one into
  each project's `test`, and each opens a run of its own suite.

## A TypeScript script, or another test runner

```ts theme={null}
// script.ts
import { openRun } from "@ambral/evals"
import { reply } from "./agent.ts"

const cases = [
  { id: "refunds/within-the-limit", message: "Please refund order 1042: $80, it arrived broken.", expected: "refund" },
  { id: "refunds/over-the-limit", message: "I want my $340 back for order 1044, it never came.", expected: "escalate" }
]

const run = await openRun({ workload: "support-agent", taskset: "refunds-scenarios" })

try {
  const reported: Array<Promise<void>> = []

  for (const { id, message, expected } of cases) {
    const scenario = run.scenario(id)
    const answer = await reply(message, scenario.headers)
    const solved = answer.includes(expected)

    reported.push(scenario.report({
      passed: solved,
      rewards: { solved: { score: solved ? 1 : 0 } },
      reasons: { solved: `Expected it to ${expected}; it replied: ${answer}` },
      task: { prompt: message, answer: expected }
    }))
  }

  await Promise.allSettled(reported)
} finally {
  await run.close()
}
```

* `scenario.report(result)` resolves once Ambral stored it, and reports go together, about once a second: collect
  them and wait for them together, not one after another.
* `run.close()` sends what is left, prints the run's id and the command that reads it, and rejects naming each scenario
  this process ran that has no result, a refused report's reason included. Close in a `finally`: a run nobody closed
  stays open.
* `run.scenario(id, { session })` takes the agent's own session id.

## A model client built once

A client built once, at module scope, can't take each scenario's headers as its defaults. Build it with the SDK's
transport instead, which adds the current scenario's headers to each call, and refuses a call made outside a scenario
rather than send it as production's.

In TypeScript, the transport is `fetch` from `@ambral/evals`:
`new OpenAI({ baseURL: process.env.AMBRAL_GATEWAY_URL, fetch })`. In a test of `@ambral/evals/vitest`, the test's
scenario is the current one. A script runs each scenario's agent within `scenario.run(() => …)`.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.