evals tell you: "given a frozen input, is the output correct against a frozen ground truth." that's necessary, but it's not enough. agents are loops. they react to what they see, generate new state, make decisions across many turns. you can't evaluate a loop by running one forward pass and grading the output.

so you simulate. and over time we ended up needing three flavors of simulation -- each one solving a problem the previous one couldn't.

flavor 1: sme replay

the simplest possible simulation. pick a real case from production -- usually one where something went wrong. seed the same input into dev. let the pipeline run. then have an sme (the human who knows what the correct behaviour is) look at every stage and grade it manually.

this sounds primitive. it is primitive. it is also, by a wide margin, the most useful one in the first month of a new failure mode.

the reason it works: the sme already knows what "right" looks like. you don't need to spec it, codify it, or build a grader for it. they look at the output and say "stage 4 should have asked for a clarification and instead drafted a final reply." that's the bug report. you fix it. one human eye finds twenty bugs an hour.

people skip this because it's "not automated." that's the wrong frame. the goal isn't automation -- the goal is finding bugs. an sme replay finds bugs immediately. a fully automated grader finds zero bugs the first time you turn it on, because the grader is mostly grading itself.

every other flavor below was built only after the sme replay loop became too expensive to run every commit. start there. graduate when you have to.

flavor 2: /run-sim -- closed-loop trace continuity

this one tests a specific thing: when a new message arrives mid-conversation, does the pipeline thread it onto the correct prior context?

the mechanics are mildly absurd and very effective:

  1. take a real shipment from sim.shipment_set (a curated table of multi-turn cases)
  2. seed the production trace + memory state into the dev database, tagged with the run's eval_run_id
  3. replay each row's webhook with force_trace_gen=true
  4. grade self-consistency on rfc_message_id -- did the pipeline land the new message on the same trace it should have?

every shipment is a block of rows in the csv. the first row is test_class=new_trace (pipeline must mint a fresh trace). every following row is test_class=incremental (must land on the same trace_id as the first).

one artifact comes out: trace_grading.json. green or red. that's the whole report.

you trigger it from a pr comment with /run-sim. same model as /run-eval -- the lowest possible bar for the team to run it.

what's lovely about this flavor: it doesn't grade content at all. it grades structure. did the pipeline preserve continuity? this is a binary check, cheap to run, and catches an entire class of bug -- silent context loss -- that content-graders are completely blind to.

if i had to recommend one new sim flavor for any agent pipeline that has multi-turn state: this one. write a structural grader before you write a content grader. structural bugs are far worse than content bugs and ten times easier to detect.

flavor 3: the final boss -- entity-agent simulation

this is the one i'm proudest of and the one with the most moving parts.

the loop:

turn = 0
state = INITIATED
while not terminal(state) and turn < max_turns:
    turn += 1
    msg      = persona.respond(pipeline_obs_prev_turn)  # turn 1 = verbatim seed
    msg_id   = inject(msg)                              # write to db + webhook
    pipeline = wait_for_agent_eval_state(msg_id)        # poll
    grade(turn, pipeline)                               # writes graded row
    state    = fsm_advance(state, classifier(pipeline)) # writes lifecycle

four ideas combine to make this work.

personas

a persona is a yaml spec for what kind of user this is. tone, knowledge level, demands, willingness to clarify, preferred channel. the persona is itself an llm -- it observes what the pipeline did last turn and responds as that user would respond, given that pipeline output.

this is the critical difference from "scripted simulation." a script gives the user's next message regardless of what the pipeline did. a persona reacts. if the pipeline asked a clarifying question, the persona answers it. if the pipeline ignored a constraint, the persona chases. you can't catch "the pipeline ignored the user's constraint" without a reactive user.

fsm lifecycle

each turn, a classifier looks at the pipeline's action and emits a transition: INITIATED → ASKED → CLARIFIED → QUOTED → BOOKED → DONE. the simulation ends when you reach a terminal state.

without an fsm you don't know when to stop. with an fsm, the conversation ends naturally -- same as the real world. and the lifecycle history itself becomes a graded artifact: did this conversation reach DONE, or did it die at QUOTED because the pipeline never followed up?

k-repeat

run the same shipment k times. grade each run independently. an agent that's correct 80% of the time looks identical to one that's correct 100% of the time if you only run it once. k-repeat is how you see the reliability cliff.

this single change reframed how we thought about agent quality. "did it pass?" became "what's the pass rate?". the eval is no longer pass/fail; it's a distribution. a regression that drops you from 95% to 75% reliability is invisible on a single run and obvious on K=10.

entropy mutation

a persistent problem with simulation: you only have so many seed cases. the fix: take a real seed, extract its canonical facts via an llm (the "entropy surface" -- what can be mutated, what must be preserved, what the invariants are), apply a mutation strategy, and persist the variant as a new shipment.

python -m simulation.entropy.cli \
    --tenant <t> --seed-uid <real_uid> \
    --strategy swap-customer-preserve-lane --variants 5

five synthetic shipments from one real one. drive them through the entity-agent loop like any other shipment. the result: a stream of plausible-but-unseen scenarios, generated by the harness, graded by the harness, at zero data-collection cost.

the entropy schema is opinionated. it knows which facts couple (a party and their downstream handler must stay consistent), which are invariant (regulatory chains, document continuity), and which are freely mutable (route, value, timing). you don't get usable synthetic data without that schema. with it, you get N variants of every seed.

the grader

the grader does the unglamorous bit. it reads agent_eval_state (same shared table the eval harness uses), compares the pipeline's output for each stage against the expected output stored on sim.shipment_message, and writes one row per (attempt × message × stage) to sim.shipment_grade.

the comparators split by field type:

  • typed comparators for structured fields: string match, numeric range, list contains, set equality. cheap, deterministic, fast.
  • llm-judge for free text (drafts, summaries, narrative outputs). slow, expensive, the only thing that works.

trying to make one comparator do both jobs is the most common failure mode i've seen in eval/sim code. typed comparators choke on free text. llm judges are wasteful and noisy on a numeric field. split them. it's the same lesson as the evals post: pick the right tool for the field.

what i actually learnt

  • start with sme replay. automation is the second step. the first step is finding bugs at all.
  • structural grading before content grading. continuity, threading, state-machine progression -- easier to test, and the bugs are worse when they happen.
  • personas, not scripts. an agent simulation without a reactive user is missing the entire point. you're not testing one forward pass; you're testing the loop.
  • k-repeat or it didn't happen. a single run masks variance. distributions, not points.
  • entropy mutation kills the data bottleneck. the seed corpus is finite; the mutated space is not.
  • typed comparators + llm-judge, split by field type. don't make one comparator do everything.
  • write the persona + fsm + grader as separate, swappable pieces. they evolve at different speeds. you'll add personas faster than you'll change the fsm, and you'll change the grader more often than either.

the simulation harness took roughly nine months to grow. nobody sat down and designed it upfront -- it accreted, each flavor solving a problem the previous flavor couldn't. that's the second-most important meta-lesson behind "use a slash command": don't try to design the whole simulation framework in advance. ship the smallest sim that catches today's bug. when today's sim stops catching tomorrow's bug, build the next flavor.