Skip to main content
How a scoring config picks the points a run replays, scores each reply, and turns those scores into each trace’s.

How a score is built

  • Points. points says where in each trace a reply is asked for, in one step: tools says which of the recorded assistant turns can be replayed, by what production called there, and take which of those are. At each point, each subject gives its reply there. Picking by tools replays only those turns, so a run costs only their replies and judges, and exports only them. points.tools sees only the recorded turn, since nothing has replied before the replay: a subject that would escalate where production didn’t is never asked there. To catch that, ask at every turn, points: { tools: null, take: all }, with a group whose turns match the call on the reply and whose criterion says when the call is due (as the ticket-agent example’s escalation group does: its escalates_to_right_team answers needless_escalation; that example samples its points), or with a scorer that grades every reply.
  • A reply’s score. The taskset scorers score each reply. fromBaseline compares it with the baseline’s, the reply the recorded trace holds at that point (did it make the same tool calls, does it say the same thing); criteria grade it on its own terms. A reply’s score is the weighted mean of the scorers that count toward it (each rewards entry, judge and criteria group, by its weight); metrics are recorded beside it without counting, unless required names one. A scorer that doesn’t apply at a point records nothing there and doesn’t count.
  • required for what must never fail. A scorer, a group or a criterion (<group>/<criterion>) named in required that scores below 1 makes its reply score 0; a criterion answered n/a doesn’t. So a required criterion is yes or no: a middle choice scores below 1, and would fail its reply too.
  • The trace’s score (episode). With no episode.rewards, a trace scores the mean of its replies’ scores, and one none of whose replies has a score has no trace score. episode.rewards.replies combines its replies’ scores by mode (mean, min, all-pass: 1 only when every scored reply scores 1, or any: 1 when one does); episode.rewards.trajectory is one judge call over the whole session; each of episode.rewards.checks scores one rule once per trace (see “Score a rule that applies only sometimes, once per trace”). Once episode.rewards names any, the trace scores the mean of those alone, so reply scorers count toward it only through replies or a check.
  • Production’s score. Production is scored by criteria groups alone: every other scorer, fromBaseline and the trajectory judge, compares a reply with the baseline’s, which production’s is. So where the config has other scorers that count, production’s per-trace score is the mean of its criteria alone, while a candidate’s includes the rest too. Compare production with a candidate on the criteria rows of a run’s summary, not the per-trace row, unless criteria groups are all that count.
  • What ambral tasksets scoring set <workload>/<name> --file <file> --check warns about. A scorer that is recorded but counts toward nothing: when episode.rewards names a scorer but not replies, each reply scorer that no check there combines, and each required entry, which can’t reach the trace’s score (a criteria group counts when checks there combine it or each of its criteria; with only some checked, the warning names the rest); and otherwise a reward, judge or criteria group of weight 0 that required doesn’t name, and no check under episode.rewards combines. Also a group’s turns that points.tools keeps from running, and a check that a subject escapes by never making a call (see “When only some moments matter”). They are warnings, not mistakes: the config still saves.

Score a rule that applies only sometimes, once per trace

A criterion with a notApplicable, or a group whose turns narrow it, scores only some replies. Per point, its mean weights a model by how often it triggered the rule: one that touched refunds 40 times counts 8 times as much as one that touched them 5 times, and a trace where the rule never applied disappears. A check scores the rule once per trace instead. The chat-agent, coding-agent, ticket-agent and state-writes examples show checks of each kind.
  • When to add one. For each criterion with a notApplicable, and each group with turns, that states a rule (“every time X, Y”), when the run replays more than one point per trace. With one point per trace, a check is its scorer’s score there, and adds nothing (the retrieval example). Under take: sample it sees only the sampled replies.
  • n/a or a filter. A filter makes no judge call, but sees only the tools a turn called (by name, any or none) and whether it is the last. A condition that takes judgment (“the reply speaks of a refund”) is the criterion’s notApplicable, which costs a judge call at each point. A check n/a in most traces is a sign a filter could save those calls.
  • mode. all-pass for a rule: one breach fails the trace. min for a graded criterion, where a trace is as good as its worst reply. mean for a degree of quality, where one weak reply among good ones lowers the trace instead of failing it. any for at least once.
  • whenNone. Keep null unless every trace must contain the action; then fail, as the coding-agent example’s done_means_tested does. Never pass: a model trained on the export learns that never acting always passes. fail works with a filter too. A rule about how a session ends (“every session ends with the ticket closed”) can be met only at the trace’s last point, where the subject replies to production’s history: for a candidate, it measures whether it ends the session where production did, not whether it would finish its own.
  • Where it counts. Under episode.rewards, it counts in the trace’s score; add replies beside it to keep the replies counting too; a trace where it has no score leaves it out of the trace’s mean, not 0, and a trace where no reward under episode.rewards has a score has no trace score. Under episode.metrics, it is recorded and counts for nothing. A check combines its scorer’s own scores, which required doesn’t change: required zeroes a reply’s score, which reaches the trace only through replies (or, with no episode.rewards, the mean of its replies). So with checks alone under episode.rewards, required changes no trace’s score; an all-pass check of the rule fails the trace instead.
  • A check or episode.replies. replies combines each reply’s whole score, every scorer together; a check, one scorer’s. With one scorer the two nearly coincide: write the check, which names the rule a trace broke.

When only some moments matter

Most configs grade every sampled turn. Where what matters happens at a few turns of a long session (a write to a record, a message sent, an escalation), a targeted run beside the main config scores only those turns. The state-writes example puts the steps together:
  • Select the turns with points: { tools: [<tool>, …], take: all }, naming only the tools whose calls are the moments you grade: a run replays, pays for and exports only the turns where production called one. A config has one points, so asks that need different selections (only the writes, and every turn) are separate runs, each with its own --scoring file.
  • Grade each action where the reply makes it, with a group per tool whose turns name it on the reply: no judge call where the reply doesn’t make the call.
  • Score each rule once per trace with a check over its group (see “Score a rule that applies only sometimes, once per trace”).
  • Fail a reply that skips the call with a calls check beside each such check, { calls: [<tool>] }, under the same episode.rewards or episode.metrics (a metric backs no reward). mode: any passes a trace whose reply made the call at one point where production did; all-pass needs it at every such point. Without one, a subject that never makes the call has no score from the group’s check, so the check can’t fail it; --check warns at such a check. A rule that leaves replies out by notApplicable rather than by turns needs its calls check just the same, and --check won’t flag it.
  • See where it ran. A run’s summary shows how often each group ran and each check scored, and how a check reads beside its calls check.
  • Run broadly now and then. The same groups under points: { take: all } ask at every turn, so they see a subject that acts where production didn’t, which the targeted run never asks.
A calls check, match or sensible_step. Use a calls check where only whether the action happens matters; match or same_tools where the reply’s calls must be production’s, whole or by tool; and sensible_step where acting a turn later, after a lookup or a question, can be right, at a judge call per point.