Skip to main content
Start score runs of a taskset, read what they found, rescore those that completed, cancel those still running, and resume those cancelled Every ambral command also takes the global flags.

ambral runs start

Start a score run of a taskset: each of its traces is replayed for each subject, and each reply scored. It scores under the taskset’s current scoring config, or --scoring’s. Production’s replies are the ones the taskset’s traces recorded, so scorers that compare a reply with those record nothing for it. A run takes minutes, longer with more subjects, traces and points, and judge calls at each: three subjects over ten short traces can take more than ten minutes. Without --wait it prints the run’s id at once; read it with ambral runs show <run> --json until its run.status is no longer running (it ends completed, failed or cancelled), and find it again with ambral runs list --taskset <workload>/<taskset>, or by a --label it was started with. A rescore of it keeps its labels Examples

ambral runs rescore

Rescore a completed run: a new run judges its replies again, every subject’s on every trace, under the taskset’s current scoring config or --scoring’s, without asking any model for a new reply. Only its judges are called and paid for; each subject’s reply cost is what the rescored run paid for those replies. Use it to try a changed config on fixed replies, or to see how far a judge agrees with itself under an unchanged one. The new run keeps the run’s labels, so ambral runs list --label finds it beside it. Its points must be ones the run replied at: under the same tools, take last, then sample by its k, then all, each covering those before it; under a run that took all, tools that match only turns its own matched Examples

ambral runs list

List the org’s score runs, newest first, a page at a time with how many there are, or a workload’s, a taskset’s, a label’s, a kind’s or a model’s, each with how it went: a replay run’s trials done, then its best candidate beside the serving model; an offline run’s pass count and mean score, and where its suite ran Examples

ambral runs show

Show a run: while it runs, how many of its trials are scored, its health (active, queued, retrying, or stalled when nothing has happened for an hour, so it won’t end by itself), each subject’s trials by where they stand, and each trial waiting to be retried; once it ends, each subject’s mean scores side by side, per trace, per scorer (each reward, metric, judge, criteria group and criterion) and per kind of point, what its replies cost and how many of its points were scored, what the judges cost, and each trial that failed. An offline run, which a suite reports: how many scenarios and runs of them it reported, how many passed by the suite’s own verdict, and their scores, each reward’s beside them Examples

ambral runs verdicts

List a run’s verdicts: each check’s result on each subject’s reply, and on each whole trace, with the judge’s verdict and reason where a judge answered, by trace, subject and point, a page at a time. Where its judge asks twice (askTwice), a verdict shows how many of the calls gave it (no 2/3), and --disagreed lists only those whose calls differed. Trials that failed are left out, as the summary’s per-point scores leave them out. Each trace is printed as tr_…:<node>, the branch the run replayed, which ambral tasksets create takes. An offline run’s: each reward its suite reported of each run of a scenario, with the reason it gave, by scenario, repeat and check, filtered by --check, --below, --scenario and --failed, each row with the trace its session was recorded under Examples

ambral runs trace

Show a run’s traces reply by reply, one or a page of those it scored: each subject’s trace score and its checks of the whole trace, then at each point what led up to it, the baseline’s reply, and each subject’s reply, with the model call that made it and each check’s result and reason. --subject keeps one subject’s replies. With --json, each check also carries the judge’s whole call: the prompt it was sent and its reply. Of an offline run: one run of a scenario, by its tr_… or the scenario’s id (with --repeat when it ran more than once), its task, each reward its suite reported with its reason, and its session’s conversation and traces under other workloads; without one, a page of its runs of scenarios; with --no-result, its sessions that made calls with its headers but have no result Examples

ambral runs export

Print a run’s results in verifiers’ episode format, one JSON object a line, which verifiers and prime-rl load from traces.jsonl. It reads them a page at a time, and prints a page only once it holds every episode the server counted, asking again for one that came back short; if one still does, it stops with an error on stderr, with --json too, that says how many it printed, and exits 1. A run still running exports the trials finished when each page was read, with a warning on stderr. So a file it wrote holds the whole run only when the run had ended and the command exits 0 Examples

ambral runs cancel

Cancel a run that is still running: it ends cancelled, and its trials not yet scoring are skipped, so they call no model. A trial already scoring is stopped within seconds, its model calls cut off; one that finished first keeps its result. Cancelling a cancelled run again changes nothing; a run that completed or failed can’t be cancelled. --subject cancels one subject the same way, and the others carry on: the run completes once they finish, and the subject’s results cover only the trials it finished. Cancelling the last subject still scoring cancels the run Examples

ambral runs resume

Resume a cancelled run, whether someone or its budget cancelled it: it runs again, scoring the trials it left unfinished, each with the retries it had left, and those it finished keep their results. It scores under the same config, and its cancelled subjects stay cancelled. It keeps its budget unless --budget gives another, which must be above what it has spent, or its next call would cancel it again. A run cancelled before Ambral kept a cancelled run’s unfinished trials can’t be resumed Examples