this is what i learnt building evals for an agentic pipeline that classifies inbound requests, extracts fields, picks who to route them to, and drafts a reply -- running across many tenants, gating every pr we merge. not a tutorial. just the parts of the journey i wish someone had told me on day one.

the embarrassing first version

the first commit of the eval system fetched a mailbox over http and dumped rows to a csv. that was the "eval system." there was no grading. you eyeballed the csv. if rows looked vaguely right, you shipped. if they looked wrong, you didn't.

the next dozen commits were a parade of fragile things: a script polling the inbox, a script writing to a local sqlite db, a script syncing from prod and trying to keep a local replica fresh. every one of these died, because keeping local state in sync with a moving production pipeline is a tax you pay forever, and you only stop paying it when you delete the local state.

the architectural shift

at some point we got tired of fighting the sync layer and did the obvious thing: stop keeping local state at all. the pipeline started writing its outputs to one shared table -- one row per processed message. the harness reads from that table. that's it. no local replica, no sync script, no "wait, this row is stale by four hours."

the corollary was even better. we added an eval_run_id column. every webhook replay carries an id, the pipeline writes it into the row, and the harness queries by that id. parallel runs stopped colliding overnight. two engineers could run evals on their own branches at the same time and never see each other's rows.

if i could only keep one idea from the entire evolution, it would be this one. an agentic pipeline writes a lot of intermediate state. don't try to mirror it. give every run a tag and read it back.

one bash command

this is the entire eval interface for the team:

./eval.sh data/emails.csv --max-time 300 --job-id 123X

that's it. one shell command. an engineer who has never touched the harness can copy-paste it, get a run, see a report.

every time someone proposed "let's build a ui for this" we did not build a ui for it. the bash script kept winning. it's grep-able, diff-able, copy-paste-able into slack, runs the same way locally and in ci, and survives every refactor of the pieces underneath. uis decay. one bash command does not.

parallel everything

the next big jump was the day we made every phase concurrent at once -- webhook replay, polling the shared table, fetching results, computing the grades. before that, a 100-row eval took around 25 minutes. after, around 3. nothing changed about what we evaluated -- only that every blocking phase moved to an async fan-out.

if your eval pipeline is sequential, your eval pipeline is wrong. evals are embarrassingly parallel. you're running the same recipe over many independent rows. anything that isn't a fan-out is leaving wall-clock on the table for no reason.

llm-as-judge, eventually

we tried everything in the embedding-similarity bucket. cosine similarity, fuzzy match, weighted hybrid scoring. they all worked just well enough to mislead us. a draft that meant the wrong thing in friendly language would score high. a draft that meant the right thing in clipped language would score low. the metric was lying.

the real shift was when we ripped all of that out and replaced it with llm-as-a-judge -- a small fast model, with a prompt that understands what "correct" means for each field type. it covered the free-text drafts first, and then the extracted-field comparisons too, once we found that even structured fields had enough fuzz around them (aliases, abbreviations, ordering) that a judge beat hand-written matchers.

the lesson: free-form text outputs do not have a numeric distance metric. they have a correctness judgement, and that judgement is itself an llm call. trying to compress that into similarity is forcing the wrong tool on the wrong job. accept that your eval pipeline contains a model in it. budget for it.

the mergeability formula (and the pendulum)

this is the one place the harness has math:

final_score  = weighted_sum(detection_f1, extraction, draft_quality,
                            agent_selection, agent_draft_quality)
penalty      = clamp(base_penalty * multiplier, 0, 1)
mergeable    = (final_score >= 0.60) AND (failure_rate <= 10%)

components are weighted (detection 35%, extraction 15%, drafts 15%, routing 20%, agent drafts 15%) and pipeline crashes apply a penalty.

then the funny part -- the pendulum:

  1. first cut: soft gates. each metric had a threshold but failing it just reduced the score.
  2. someone shipped a regression that dropped one metric off a cliff and the overall score stayed green. embarrassing.
  3. we flipped to hard gates: any single gate failing → no merge. the score became informational only.
  4. weights got tweaked across maybe twelve commits as we learnt where the false positives lived. different parts of the pipeline pulled in different directions.

the lesson: a single number is a lie. an agentic pipeline has heterogeneous skills (detection vs. routing vs. drafting) and you can't average them into one number without hiding the failure. hard gates per skill are honest. the overall score is just a sticker on the box.

tiered datasets

once the harness was fast and the gate was honest, we sliced datasets by speed:

  • axon_evals_prime_2.csv -- 30 rows, per-pr, deterministic
  • axon_evals_nightly.csv -- 50 rows, nightly on dev, sourced from prod's last 30 days
  • axon_evals_release.csv -- 50 rows, weekly + pre-prod-bump, full closed-loop chains

the trick is that every tier uses the same harness. swap the csv path, don't swap the runner. a per-pr eval and a release eval differ in cost, not in code. that means an engineer who knows how to run the per-pr eval already knows how to run the release eval. zero context switch.

github actions, or: how to make evals a team sport

the last unlock wasn't code. it was a github action that listens for /run-eval comments on pull requests.

/run-eval --dataset eval_data/axon_evals_prime_2.csv

the bot picks up the comment, kicks off the harness, waits, posts the mergeability report back as a pr comment, and flips a status check. green = mergeable. red = read the report.

before this, every engineer was supposed to run evals locally before merging. most people didn't. the ones who did ran them inconsistently, on different datasets, with no shared record of the result -- so even when someone did the right thing, there was no way to track it. evals were a virtue tax, and virtue taxes don't get paid. with the slash command, anyone on the team runs the eval on their own branch in two characters of typing, the bot writes the result back as a pr comment, and the status check is visible to every reviewer. shipping safely stopped being a personal virtue and became a default.

if you build an eval harness and don't expose it as a pr-comment slash command, you've left 80% of its value on the floor. the harness is for the team, not for the person who built it.

what i actually learnt

  • don't keep local state. give every run an id, write to one shared table, read by id. saves a year of sync bugs.
  • one bash command beats a ui. every time.
  • evals are embarrassingly parallel. if yours aren't, fix that before anything else.
  • llm-as-judge for free text. fuzzy similarity is a trap that smells like a metric.
  • hard gates per skill, not a single number. averages hide regressions.
  • slash commands on prs. an eval is only worth what the least disciplined engineer on the team uses. a pr comment is the lowest possible bar.

the harness ended up being about 3,000 lines of python, a docker-compose, two yaml workflows, and a bash script. it gates every pr we merge. it would have taken months to build all at once and we never tried -- it grew, one commit at a time, fixing the thing that hurt that day. that's probably the only way these things ever get built.