Skip to main content
Replay runs and offline runs, and the primitives Ambral is built from: workloads, sessions, traces, tasksets, scoring configs and runs.

Replay runs and offline runs

Ambral evaluates an agent two ways, and most agents should have both.
  • A replay run (also called a shadow eval) is Ambral’s. It replays the agent’s recorded production sessions to each candidate: at each point the scoring config picks, the candidate writes the next reply to the session as production lived it, and the config scores that reply. It needs no environment, no tools and no suite, so it is where to start, and the fastest way to find failure modes, try a prompt, and compare models on real traffic.
  • An offline run is the team’s own suite’s. The suite runs each scenario whole, in its own environment, with the candidate making every call, and reports each scenario’s scores to Ambral as it runs.
A replay run’s signal is directional. The candidate writes one reply at a time to production’s history, never to its own earlier replies, so it is scored only where production’s model replied. Anything that shows only on the candidate’s own path is invisible to it: a model that would loop on a tool for 50 steps, where production’s session finished in 3, is scored on those 3 replies, each of which may look fine. Drifting off the task, recovering from its own mistake, and taking another route to the same goal are invisible too. So neither replaces the other, and neither replaces real traffic. Replay runs show where to look: the failure modes, and the sessions that split models, which become scenarios. Offline runs and a canary both see whole sessions on the candidate’s own path: a suite before any user does, on cases production seldom brings, and a canary, the candidate serving a small share of production’s sessions, on real traffic with no suite, its share bounding the risk. Neither is a gate the other must pass first. A suite’s checks are a good source for a replay config’s criteria, since they say what the team already counts as a failure.

Primitives

  • Workload: one kind of task the agent repeats (support-agent), and the model that serves it, which can change behind the Gateway with no change to the agent. Its pool is the models a run may score for it (ambral workloads pool add|remove), and its primary is the pool model that serves it (ambral workloads set-primary).
  • Session: each time the agent does that task, a chat with a user or one pass of an automated workflow, however many model calls it takes, recorded, evaluated and routed as a whole.
  • Trace: one recorded session (tr_…), stored under its workload.
  • Taskset: an eval, named <workload>/<taskset> (for example support-agent/refunds). A replay taskset holds recorded traces picked by hand, with its own scoring config; an offline taskset holds the scenarios a suite reports to it (support-agent/refunds-scenarios), which its first run creates.
  • Scoring config: a YAML file saying where a run asks for replies and how it scores them. Each save is the taskset’s next scoring version.
  • Preset: a ready-made scoring config to start one from: strict, balanced, trajectory or criteria.
  • Run: a replay run scores a taskset for its subjects, replaying each trace and scoring the replies the way the config says. An offline run is one run of a suite on one model: what the suite reported of each scenario.
  • Scenario: a case of a suite, by the suite’s own stable id (refunds/over-the-limit). Each time it runs in a run is a repeat, with a result (what the suite reported: passed, rewards, reasons) and calls (the agent’s model calls the Gateway recorded).
  • Trial: one trace replayed for one subject. A trial whose model or judge call fails is retried a few times; one that still fails has no scores, and its run ends failed.
  • Health: how a running run is doing: active (trials are being scored), queued (its trials wait their turn), retrying (some failed and wait to be retried) or stalled (nothing has happened for an hour, so it won’t end by itself). ambral runs show prints it.
  • Subject: what a run scores: production, what the agent sent, or a model of the workload’s pool, which writes its own reply to the recorded session so far.
  • Point: a recorded assistant turn where the config asks for a reply.
  • Baseline: the reply the recorded trace holds at each point, which is production’s reply there.
  • Criteria: statements you write that a judge model grades each reply against, on its own terms, never against the baseline’s.