- A reward is from 0 to 1. One outside it is refused before it is sent. A suite that scores out of 10 divides by 10; a figure with no bound (a count, a cost) is a metric, not a reward.
- Give a reason with every reward. A reason is what
ambral runs verdictsprints beside a failure, and what reading the failures for their failure modes starts from. A score without one says only that something went wrong. - Send
passedwhen the suite has a verdict of its own. Where it doesn’t, the run’s results stand mean scores in for pass counts, andambral runs verdicts --failedis refused: use--below 1. - Send the task. It is what
runs traceshows of a scenario, and what tells a changed case from the same one: a changed task is a new revision, whichtasksets resultsmarks. Without it, every run of a scenario reads as the same revision, whatever changed. In Vitest,scenario.task(data); under the pytestconftest.py,scenario.task = …; in a script, the result’stask. verifiers reports the episode’s task, and Harbor the task’s checksum, by themselves. Each process sends it once per scenario, then its id alone.
Report your own offline suite
What a result holds
What a suite reports of each scenario of an offline run, and the environment the SDKs read.
What a suite reports of each scenario of an offline run, and the environment the SDKs read.