Skip to main content
Grade only the moments that change something, each message to the tenant and each write to the record, and don’t let a model pass by never making them. It is one targeted run beside the workload’s main scoring, not all of it. Based on the criteria preset. A property-management agent that works tenant requests on its own: it reads the lease (get_lease), messages the tenant (message_tenant), and writes what it learned to the tenant’s record (update_tenant_record), which the rest of the business reads. Most of a session is lookups, so the run replays only the turns where production called one of the two tools that change something outside the session: a message the tenant reads, and a write to the record. A run pays for those replies alone, and exports only them. To grade only the writes, select only the write tool. Each group grades its own tool’s calls in the reply, so no judge is called where the reply doesn’t make the call. A reply that skips the call isn’t graded there at all, so each check over a group has a calls check beside it, which fails that reply with no judge call. The trace scores the mean of its checks that have a score. What it can’t see: a subject that would message or write where production didn’t is never asked there, and a lookup before the write isn’t graded. A broader run, points: { take: "all" } with the same groups, asks at every turn: run it now and then, or when a model’s targeted scores look too good, to see what this one skips. An example to learn from, not a starting point: start a config from ambral tasksets scoring preset, and take from this one the choices that fit your agent.