Skip to main content
Match the baseline turn by turn, keep to a policy it must never break, and hand to a person what the policy says to. Based on the balanced preset. A support agent in a live chat: it answers customers and changes their orders under a refund policy in its system prompt. It grades turn by turn against the baseline, as the balanced preset does (a judge asks whether each reply is a sensible step, and whether the last one is the right answer), and adds criteria for what a good support session must do whatever the baseline did: confirm before changing an order, keep to the policy, explain it plainly, and never invent a date. Breaking the policy fails the reply outright, and checks score each rule that applies only sometimes once per trace. A second group checks what a judge comparing with the baseline seldom catches, for this agent: no claim of work not done, no instruction followed from text someone else wrote into the conversation, and, since it can hand a chat to a person, a handoff exactly when the policy calls for one. With take: sample a missed handoff at a turn that isn’t sampled is seen by nothing: take all where missed handoffs matter. It fits an agent in a live chat, where each reply is a step the customer sees and answers. A support agent that answers emails or tickets on its own, looking up what each one needs and then acting, is judged by what it changed instead: see ticket-agent.yaml. An example to learn from, not a starting point: start a config from ambral tasksets scoring preset, and take from this one the choices that fit your agent.