criteria preset.
A property-management agent that works tenant requests on its own: it reads the lease (get_lease), messages the
tenant (message_tenant), and writes what it learned to the tenant’s record (update_tenant_record), which the rest
of the business reads.
Most of a session is lookups, so the run replays only the turns where production called one of the two tools that
change something outside the session: a message the tenant reads, and a write to the record. A run pays for those
replies alone, and exports only them. To grade only the writes, select only the write tool. Each group grades its
own tool’s calls in the reply, so no judge is called where the reply doesn’t make the call. A reply that skips the
call isn’t graded there at all, so each check over a group has a calls check beside it, which fails that reply
with no judge call.
The trace scores the mean of its checks that have a score.
What it can’t see: a subject that would message or write where production didn’t is never asked there, and a
lookup before the write isn’t graded. A broader run, points: { take: "all" } with the same groups, asks at every
turn: run it now and then, or when a model’s targeted scores look too good, to see what this one skips.
An example to learn from, not a starting point: start a config from ambral tasksets scoring preset, and take
from this one the choices that fit your agent.