ambral runs start
--scoring’s. Production’s replies are the ones the taskset’s traces recorded, so scorers that compare a reply with those record nothing for it. A run takes minutes, longer with more subjects, traces and points, and judge calls at each: three subjects over ten short traces can take more than ten minutes. Without --wait it prints the run’s id at once; read it with ambral runs show <run> --json until its run.status is no longer running (it ends completed, failed or cancelled), and find it again with ambral runs list --taskset <workload>/<taskset>, or by a --label it was started with. A rescore of it keeps its labels
Examples
ambral runs rescore
--scoring’s, without asking any model for a new reply. Only its judges are called and paid for; each subject’s reply cost is what the rescored run paid for those replies. Use it to try a changed config on fixed replies, or to see how far a judge agrees with itself under an unchanged one. The new run keeps the run’s labels, so ambral runs list --label finds it beside it. Its points must be ones the run replied at: under the same tools, take last, then sample by its k, then all, each covering those before it; under a run that took all, tools that match only turns its own matched
Examples
ambral runs list
Examples
ambral runs show
active, queued, retrying, or stalled when nothing has happened for an hour, so it won’t end by itself), each subject’s trials by where they stand, and each trial waiting to be retried; once it ends, each subject’s mean scores side by side, per trace, per scorer (each reward, metric, judge, criteria group and criterion) and per kind of point, what its replies cost and how many of its points were scored, what the judges cost, and each trial that failed. An offline run, which a suite reports: how many scenarios and runs of them it reported, how many passed by the suite’s own verdict, and their scores, each reward’s beside them
Examples
ambral runs verdicts
askTwice), a verdict shows how many of the calls gave it (no 2/3), and --disagreed lists only those whose calls differed. Trials that failed are left out, as the summary’s per-point scores leave them out. Each trace is printed as tr_…:<node>, the branch the run replayed, which ambral tasksets create takes. An offline run’s: each reward its suite reported of each run of a scenario, with the reason it gave, by scenario, repeat and check, filtered by --check, --below, --scenario and --failed, each row with the trace its session was recorded under
Examples
ambral runs trace
--subject keeps one subject’s replies. With --json, each check also carries the judge’s whole call: the prompt it was sent and its reply. Of an offline run: one run of a scenario, by its tr_… or the scenario’s id (with --repeat when it ran more than once), its task, each reward its suite reported with its reason, and its session’s conversation and traces under other workloads; without one, a page of its runs of scenarios; with --no-result, its sessions that made calls with its headers but have no result
Examples
ambral runs export
traces.jsonl. It reads them a page at a time, and prints a page only once it holds every episode the server counted, asking again for one that came back short; if one still does, it stops with an error on stderr, with --json too, that says how many it printed, and exits 1. A run still running exports the trials finished when each page was read, with a warning on stderr. So a file it wrote holds the whole run only when the run had ended and the command exits 0
Examples
ambral runs cancel
cancelled, and its trials not yet scoring are skipped, so they call no model. A trial already scoring is stopped within seconds, its model calls cut off; one that finished first keeps its result. Cancelling a cancelled run again changes nothing; a run that completed or failed can’t be cancelled. --subject cancels one subject the same way, and the others carry on: the run completes once they finish, and the subject’s results cover only the trials it finished. Cancelling the last subject still scoring cancels the run
Examples
ambral runs resume
--budget gives another, which must be above what it has spent, or its next call would cancel it again. A run cancelled before Ambral kept a cancelled run’s unfinished trials can’t be resumed
Examples