> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ambral.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> To set Ambral up in a project, run `npx ambral@latest skill install` in the project's root. It installs Ambral's skill, matched to the CLI's version, and the skill says how to do everything these pages describe with the `ambral` CLI. Prefer it to working from these pages. Never ask anyone for an Ambral API key, and never handle one: the person creates it in the web app and stores it where their app reads its secrets.

# Python: ambral-evals

> Report a pytest suite, a script, a verifiers eval or a Harbor job to Ambral as an offline run.

Report a pytest suite, a script, a verifiers eval or a Harbor job to Ambral as an offline run.

In Python, give a model client built once `httpx.Client(transport=Transport())`, `Transport` from
`ambral_evals.httpx` (`AsyncTransport` for an `httpx.AsyncClient`), and run each scenario's agent within
`with scenario.current():`.

## pytest

`pip install ambral-evals`, which needs Python 3.10 or later. The Python SDK has no pytest plugin: this `conftest.py`
reports each test that takes the `scenario` fixture, its id the test's node id
(`test_refunds.py::test_refund[over-the-limit]`). Copy it into the suite's directory and set `WORKLOAD` and `TASKSET`:

```python theme={null}
"""Reports each test that takes the `scenario` fixture to Ambral, as one run of a scenario whose id is the test's."""

from __future__ import annotations

import os
import uuid
from collections.abc import Generator, Iterator, Mapping
from typing import Any

import pytest

from ambral_evals import Result, Reward, Run, Scenario, open_run

WORKLOAD = "support-agent"
TASKSET = "refunds-scenarios"

CI_RUN_IDS = ("GITHUB_RUN_ID", "CI_PIPELINE_ID", "CIRCLE_WORKFLOW_ID", "BUILDKITE_BUILD_ID")
"""The variables the SDK keys a run by on a CI, so every job of the CI's workflow run shares it."""

_CALLED = pytest.StashKey[pytest.TestReport]()


def pytest_configure(config: pytest.Config) -> None:
    # pytest-xdist's workers inherit the controller's environment, so with one key they open one run.
    keyed = any(os.environ.get(name) for name in ("AMBRAL_EVAL_KEY", *CI_RUN_IDS))
    if not hasattr(config, "workerinput") and not keyed:
        os.environ["AMBRAL_EVAL_KEY"] = f"pytest/{uuid.uuid4()}"


@pytest.hookimpl(wrapper=True)
def pytest_runtest_makereport(item: pytest.Item) -> Generator[None, pytest.TestReport, pytest.TestReport]:
    report = yield
    if report.when == "call":
        item.stash[_CALLED] = report
    return report


@pytest.fixture(scope="session")
def ambral_run() -> Iterator[Run]:
    with open_run(WORKLOAD, TASKSET) as run:
        yield run


class Case:
    """A test's run of its scenario: the headers its agent's model client sends, the scenario's task, and the scores
    the test gives."""

    def __init__(self, scenario: Scenario) -> None:
        self.scenario = scenario
        self.headers = dict(scenario.headers)
        self.task: Mapping[str, Any] | None = None
        """The scenario's task, as verifiers' `TaskData` holds one: its prompt, what it expects, any keys of yours."""
        self.rewards: dict[str, Reward] = {}
        self.reasons: dict[str, str] = {}

    def score(self, name: str, score: float, reason: str | None = None) -> None:
        """Scores the scenario on reward `name`, from 0 (fails) to 1 (passes), with why."""
        self.rewards[name] = {"score": score}
        if reason is not None:
            self.reasons[name] = reason


@pytest.fixture
def scenario(ambral_run: Run, request: pytest.FixtureRequest) -> Iterator[Case]:
    case = Case(ambral_run.scenario(request.node.nodeid))
    yield case
    called = request.node.stash.get(_CALLED, None)
    if called is None or called.skipped:
        case.scenario.drop()
        return
    result: Result = {"passed": called.passed, "rewards": case.rewards, "reasons": case.reasons}
    if case.task is not None:
        result["task"] = case.task
    if called.failed:
        result["errors"] = [{"type": "TestFailed", "message": called.longreprtext[-4000:]}]
    # Queued, not awaited: the run's close, when the session ends, fails naming any report Ambral refused.
    case.scenario.send(result)
```

A test hands the agent `scenario.headers`, sets the scenario's task, and scores it:

```python theme={null}
"""Refund requests the support agent must handle by its policy, each a scenario of the suite."""

from __future__ import annotations

from typing import TYPE_CHECKING

import pytest

from agent import reply

if TYPE_CHECKING:
    from conftest import Case

CASES = {
    "within-the-limit": ("Please refund order 1042: $80, it arrived broken.", "refund"),
    "at-the-limit": ("Refund order 1043, $250. Wrong size.", "refund"),
    "over-the-limit": ("I want my $340 back for order 1044, it never came.", "escalate"),
    "split-to-dodge-the-limit": ("Refund $200 for order 1045 now, and another $200 for it tomorrow.", "escalate"),
}


@pytest.mark.parametrize(("message", "expected"), CASES.values(), ids=CASES.keys())
def test_refund(scenario: Case, message: str, expected: str) -> None:
    scenario.task = {"prompt": message, "answer": expected}
    answer = reply(message, scenario.headers)
    solved = expected in answer.lower()
    scenario.score("solved", 1 if solved else 0, f"Expected it to {expected}; it replied: {answer[:500]}")
    assert solved
```

* Give each parametrized case an id of its own (`ids=`), not the default index, so the scenario's id stays stable. A
  node id is relative to pytest's rootdir: run the suite from the same directory each time.
* Under pytest-xdist (`pytest -n 4`), the workers share one key, so they report to one run. On GitHub Actions, GitLab
  CI, CircleCI and Buildkite that is the CI's id for the workflow run, which every job shares; elsewhere the controller
  mints one per invocation (on Jenkins or Azure Pipelines, set `AMBRAL_EVAL_KEY` to the pipeline run's id to share one
  run across jobs).
* A skipped test, or one whose setup failed, reports nothing. The test file imports `Case` only for its types, so it
  runs under `--import-mode=importlib` too. The fixture needs pytest 8 or later.
* pytest captures the line the SDK prints as the run closes, which names the run. Run with `-rP` to read it, under
  `Captured stdout teardown`, once per pytest-xdist worker; or find the run with
  `ambral runs list --taskset <workload>/<taskset>`, which lists the newest first.

## A Python script, or another harness

```python theme={null}
"""Runs each case of the suite, reports each to Ambral as a scenario, and closes the run."""

from __future__ import annotations

from agent import reply
from ambral_evals import open_run

CASES = {
    "refunds/within-the-limit": ("Please refund order 1042: $80, it arrived broken.", "refund"),
    "refunds/over-the-limit": ("I want my $340 back for order 1044, it never came.", "escalate"),
}

REPEATS = 2


def main() -> None:
    with open_run("support-agent", "refunds-scenarios") as run:
        for id, (message, expected) in CASES.items():
            for _ in range(REPEATS):
                scenario = run.scenario(id)
                try:
                    answer = reply(message, scenario.headers)
                except Exception as error:
                    scenario.send({"passed": False, "errors": [{"type": type(error).__name__, "message": str(error)}]})
                    continue
                solved = expected in answer.lower()
                scenario.send(
                    {
                        "passed": solved,
                        "rewards": {"solved": {"score": 1 if solved else 0}},
                        "reasons": {"solved": f"Expected it to {expected}; it replied: {answer[:500]}"},
                        "task": {"prompt": message, "answer": expected},
                    }
                )


if __name__ == "__main__":
    main()
```

* `scenario.send(result)` queues the result and returns at once; `scenario.report(result)` waits until it is stored.
  The end of the `with` block closes the run, and raises naming each scenario with no result.
* `run.scenario(id, session=...)` takes the agent's own session id.

## verifiers

Run the eval with `ambral-verifiers`, naming the suite, then `vf-eval`'s own arguments:

```sh theme={null}
ambral-verifiers support-agent/refunds-scenarios @ eval.toml
```

The eval's agent seat calls the Gateway (`base_url`, the Gateway's `/v1` base URL), naming the run's header in its own
client:

```toml theme={null}
model = "ambral/auto"

[env.agent.client]
type = "eval"
base_url = "https://api.ambral.ai/v1"
api_key_var = "AMBRAL_API_KEY"
headers_from_env = { x-ambral-eval = "AMBRAL_EVAL_HEADER" }
```

* Keep `type = "eval"`: verifiers refuses a seat's client that doesn't name its type.
* The model is `AMBRAL_EVAL_MODEL`. verifiers sends each rollout's id as its session, and reports the episode's task,
  so nothing else is needed.
* verifiers has no verdict of its own, so its runs read by their scores.
* A scenario's id is its task's `id`, else its `name`, else its index in the taskset, which holds only while the
  taskset's order does: give each task an `id`.
* Only the agent's seat names the header, in its own client. An episode in which more than one seat sends it isn't
  reported.
* Repeats are `num_rollouts` (`-r`); concurrency is `max_concurrent` (`-c`), 128 unless set.

## Harbor

```sh theme={null}
harbor run --plugin ambral_evals.harbor:Report \
  --plugin-kwarg workload=support-agent --plugin-kwarg taskset=terminal-scenarios ...
```

The agent class sends `headers(self.context_id)`, from `ambral_evals.harbor`, on each of its model calls. Each trial is
one run of its task, whose name is the scenario's id, and the plugin reports the task's checksum, so a changed task is
a new revision. Harbor has no verdict of its own, so its runs read by their scores. The run's key is the job's id,
which a resumed job keeps. Repeats are `--n-attempts`; concurrency `--n-concurrent`. The plugin also takes `model`,
`key` and `source` as `--plugin-kwarg`s.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.