Skip to main content
What a suite reports of each scenario of an offline run, and the environment the SDKs read.
  • A reward is from 0 to 1. One outside it is refused before it is sent. A suite that scores out of 10 divides by 10; a figure with no bound (a count, a cost) is a metric, not a reward.
  • Give a reason with every reward. A reason is what ambral runs verdicts prints beside a failure, and what reading the failures for their failure modes starts from. A score without one says only that something went wrong.
  • Send passed when the suite has a verdict of its own. Where it doesn’t, the run’s results stand mean scores in for pass counts, and ambral runs verdicts --failed is refused: use --below 1.
  • Send the task. It is what runs trace shows of a scenario, and what tells a changed case from the same one: a changed task is a new revision, which tasksets results marks. Without it, every run of a scenario reads as the same revision, whatever changed. In Vitest, scenario.task(data); under the pytest conftest.py, scenario.task = …; in a script, the result’s task. verifiers reports the episode’s task, and Harbor the task’s checksum, by themselves. Each process sends it once per scenario, then its id alone.

The SDK’s settings

The SDK reads its settings from the environment: