29 of 34 passing agent runs had a blocker the hidden tests never saw

2026-10-16 · 13 min read

Contents

Assumes you've run a coding agent on a real task and read the diff it left behind.

TL;DR — A Go transfer function written by a local model passed 15 of 15 hidden core tests, 12 of 12 detail tests and 7 of 7 mutants. Cancel the request context between its two saves and 4.00 EUR disappears. To find that kind of bug in agent output at scale:

Review each run blind: task, original README, diff, final tree, mechanical score, and nothing that names the model or harness.
Score overall as a merge decision, never an average, with mechanical caps: a reproduced money loss is a blocker, and any blocker caps overall at 3.
The reviewer writes and runs a short probe test for every failure path the task names. Reasoning about a probe instead of running it gets the review rejected.
Calibrate a cheap reviewer against an expensive one on the same packets. Here the rules moved the scores, not the model.

As of the last review filed on 15 October, 69 of 75 reviews carried at least one blocker. 34 of the 75 runs had a PASS verdict, and 29 of those still carried one.

A 100-point Go transfer that loses money#

The task: a ledger transfer in Go, all or nothing even when the account store fails part-way, idempotent per key, safe under concurrency. The run: the pilot of an agent study on my 128 GB box, Ornith-1.5-35B Q8 under Claude Code. Its mechanical score:

 "verdict": "PASS",
 [...]
  "hidden_core": "15of15",
  "hidden_detail": "12of12",
  "mutants": "7of7",
  "mermaid": "ok",
  [...]
  "points": "100"

Here is the rollback, from the final tree:

// restore writes an account back to its original snapshot after a failed multi-save. It is best
// effort: a store that already failed cannot be rolled back further, and the per-account locks are
// held, so no other mutation can be waiting on a snapshot we do not have.
func (s *Service) restore(ctx context.Context, id AccountID, a Account) {
	_ = s.store.Save(ctx, a)
}

Two things are wrong, and the comment explains neither. The error goes to _. And the rollback runs on the caller's ctx, the same context that just made the second save fail. The store in this project honours ctx.Err() on every call, so a cancelled request fails the credit and then fails the restore. A probe is a test that walks into exactly that:

package ledger_test

import (
	"context"
	"testing"

	"acme.local/ledger"
)

// cancelAfterFirstSave cancels the request context once the first Save succeeds:
// the client hung up between the debit and the credit.
type cancelAfterFirstSave struct {
	*ledger.MemoryStore
	cancel context.CancelFunc
	saves  int
}

func (s *cancelAfterFirstSave) Save(ctx context.Context, a ledger.Account) error {
	err := s.MemoryStore.Save(ctx, a)
	if s.saves++; s.saves == 1 && s.cancel != nil {
		s.cancel()
	}
	return err
}

func TestProbeC_CancelBetweenSaves(t *testing.T) {
	store := &cancelAfterFirstSave{MemoryStore: ledger.NewMemoryStore()}
	svc := ledger.NewService(store, &ledger.MemoryAuditLog{}, ledger.SystemClock{})
	bg := context.Background()
	from, _ := svc.OpenAccount(bg, "alice", ledger.EUR)
	to, _ := svc.OpenAccount(bg, "bob", ledger.EUR)
	svc.Deposit(bg, from.ID, ledger.NewMoney(1000, ledger.EUR))

	ctx, cancel := context.WithCancel(bg)
	store.saves, store.cancel = 0, cancel
	_, err := svc.Transfer(ctx, from.ID, to.ID, ledger.NewMoney(400, ledger.EUR), "k1")

	a, _ := svc.Balance(bg, from.ID)
	b, _ := svc.Balance(bg, to.ID)
	t.Logf("err=%v from=%s to=%s", err, a, b)
	if a.Minor+b.Minor != 1000 {
		t.Fatalf("money lost: total %d, want 1000", a.Minor+b.Minor)
	}
}

I ran it on a scratch copy of the run's final tree. The agent's own suite passes go vet and go test -race on that copy. The probe doesn't:

$ go test -count=1 -run ProbeC -v .
=== RUN   TestProbeC_CancelBetweenSaves
    probe_test.go:40: err=transfer ACC-000002: context canceled from=6.00 EUR to=0.00 EUR
    probe_test.go:42: money lost: total 600, want 1000
--- FAIL: TestProbeC_CancelBetweenSaves (0.00s)
FAIL
FAIL	acme.local/ledger	0.002s
FAIL

Alice is down 4.00 EUR. Bob never got it. The caller sees context canceled, a perfectly ordinary error, with no hint that a debit stayed behind. Swapping ctx for context.WithoutCancel(ctx) in restore turns the probe green (from=10.00 EUR to=0.00 EUR); the swallowed error is still there for the next probe to find.

Part 13 found the same cancelled-context bug in a different run. The original review of this run recorded the same loss, "source 10.00 -> 6.00 EUR, destination unchanged", and a second blocker: the idempotency check runs only under the two account locks, so one key reused at the same time on two unrelated account pairs moved money twice in 134 of 300 probe runs. The README the agent wrote promises that two concurrent calls with the same key "move money exactly once". For the same pair, they do.

Hidden tests check what the task author thought to check. The rest of this post is the setup that checks the other part.

Every archived run becomes one directory, named by the first ten hex characters of sha256(run id). The map from blind id back to the run lives in a file reviewers never read. The packet holds:

TASK.txt: the prompt, verbatim.
README.md and docs/ from the baseline tag, not from the final tree. The reviewer judges against the conventions the team wrote. If the agent edited the README, that edit is in the diff, where it can be judged too.
change.diff: baseline to the final working tree, including uncommitted and untracked files.
final/: the tree, minus .git, build output and harness debris.
score.json: verdict, hidden tests, mutants, wall time, tool calls. No names.
CLAIMS.md: what the agent said it did, for the honesty pass.

From prepare-review.sh:

# conventions as the team wrote them (baseline), not as the agent may have edited them (its edits are in the diff)
( cd "$WS" && git show bench-baseline:README.md ) > "$P/README.md" 2>/dev/null || rm -f "$P/README.md"
[...]
( cd "$WS" && git add -A -N . 2>/dev/null; git diff --no-color bench-baseline -- . ':!*.sum' ':!vendor-docs' ':!vendor-sources' ) > "$P/change.diff" || true
# harness debris names the model/harness (run-1.log is the agent's own stdout log): keep packets blind.
[...]
rsync -a --exclude .git --exclude target --exclude node_modules --exclude .zig-cache --exclude zig-out \
  --exclude __pycache__ --exclude 'run-*.log' --exclude .claude --exclude .opencode --exclude .pi \
  --exclude vendor-docs --exclude vendor-sources \
  "$WS/" "$P/final/"
grep -rIl -i -E 'opencode|claude|ornith|qwen|gpt-oss|laguna|flash-next|strata|llama' "$P" 2>/dev/null \
  | sed "s#^$P/#  blind check: names a model/harness: #" || true

git add -A -N is the quiet part: intent-to-add puts untracked files into git diff, so a run that never committed still has a complete diff. The last two lines are the blind check. Agents sign their work everywhere, from .claude/ and .opencode/ directories to a run log that is the agent's own stdout, and the grep names each file that slipped through.

The rubric: overall is a merge decision#

Six dimensions, each 1 to 5: correctness beyond the tests, concurrency, errors and API shape, the agent's own tests, conventions and scope, readability. Every score below 5 needs a file:line finding. "Do not reward length" is in the rubric verbatim, because reviewers do.

overall is not their mean. It is the merge decision a senior reviewer on the team would make:

overall	decision
5	merge as is
4	merge after nits
3	needs changes
2	major rework
1	reject

Two caps apply mechanically before any judgement. A reproduced money loss or double move from probes a, c, d or e sets correctness to 2 or lower and is a blocker. Any blocker caps overall at 3, however tidy the lock ordering.

Probes a to g: failure paths, run as tests#

For the ledger task (Java, Go and Zig versions) the procedure lists seven probes. Probes a and f, and the save-failure leg of e, repeat cases the hidden suites already test; b, c, d, g and the other legs of e go where they don't. The reviewer writes each one as a scratch test, under about 40 lines, runs it, and reports reproduced, not reproduced, or not applicable, with the observed balances:

a. The second save fails after the first succeeded. Are balances restored? Is there a rollback at all?
b. The rollback itself fails. Is that signalled to the caller or swallowed?
c. The request context is cancelled between the two saves (Go). Does the rollback still run?
d. Same idempotency key, two concurrent calls on different account pairs. Money moved once?
e. Same key, sequential retry after a failure at each step: save, audit append, key record, or out-of-memory in Zig.
f. Blank key, same-account transfer, zero or negative amount: the typed error the README asks for?
g. Waiters on an in-flight key when the first attempt fails: released within a 2 s deadline, or hung?

The recorded results read like a bank's incident log, from runs that passed every hidden core test. Zig, 100 points, an undo whose errors go to catch {}: probe b returned StoreUnavailable "with from=600 to=0, undo errors swallowed". Java, a rollback inside catch (Exception ignored) {}: "from=70.00 to=0.00 with suppressed=0, so 30.00 vanished". Different models, languages and harnesses; the same two saves with nothing safe between them.

"Assessed by reading" is a rejected review#

The procedure says it in one line: probes must be run, not reasoned about, and a review whose summary says a probe was "assessed by reading" is rejected and redone. The summary has to list each probe with its command and the observed values.

The pilot code is the argument for that rule. Read restore with its comment and it sounds deliberate: best effort, locks held, nothing waiting. The comment on the key map says transfers "serialize on the locks of the accounts involved, so a retry is never in flight alongside the original". Both are fluent. Reading the code means reading the agent's explanation of it, and agents explain confidently. A probe has no opinion about the comment.

The rule also had to be precise about what counts as handled. The first probe rules didn't say whether a rollback failure that is reported to the caller still counts as a loss, and reviews disagreed: some called it a blocker, others a major. On 13 October the rule split it:

Rollback error swallowed (_ =, catch {}, original error returned with no trace): blocker, correctness 2 or lower.
Rollback failure reported to the caller, wrapped, joined, suppressed, even as bare %v text: major, correctness 3 or lower, not blocking.

That is rubric v1p2. It was re-applied from the probe results already recorded, without re-running any code. In 5 of the 27 earliest runs a blocker became a major; 3 overall scores went up by one.

The rules moved the scores, not the reviewer model#

The first 27 reviews were done by Opus 5.5 under rubric v1, before the probes were mandatory. One pilot review used about 100k tokens of my Claude usage, against an estimate of 10 to 40k. That is one review, not a measured average, but it was enough to go looking for a cheaper reviewer.

Calibration ran on three packets Opus had already reviewed:

reviewer	rubric	overall on the 3 packets
Opus 5.5	v1	4 / 3 / 1
Sonnet 5.5	v1 + probes	3 / 2 / 1
Haiku 5.5	v1 + probes	3 / 2 / 1

Same blockers; every Opus blocker found and reproduced. The cheap reviewers were one point stricter exactly where a probe lost money, code Opus had flagged as a major. Then Opus reviewed two of those packets under the probe rules: 3 and 2. Haiku became the routine reviewer, Sonnet the tiebreaker.

All 27 Opus reviews were redone under the new rules before any ranking used them, and the two scales never share a ranking. Against the current v1p2 scores, 14 of the 27 dropped by one point, 13 stayed, none went up. Nine 4s became three. The top of the scale compressed because nearly every run had a reproduced money-loss path.

Which runs get reviewed, and which ones that hides#

Reviewing everything costs too much, so a selector picks every run with 70 points or more, plus a safety net per arm × task × harness (an arm is a model plus its run configuration). The core of select-for-review.py:

pick = {}
for arm, pts, rid, v in runs:
    if pts >= 70:
        pick[rid] = f"points {pts:g} >= 70 ({v})"
by_arm = {}
for arm, pts, rid, v in runs:
    by_arm.setdefault(arm, []).append((pts, rid, v))
for arm, lst in by_arm.items():
    if all(p < 70 for p, _, _ in lst):
        p, rid, v = max(lst)
        if p >= 40:
            pick[rid] = f"best of {arm[0]}/{arm[1]}/{arm[2]}/{arm[3]} at {p:g} (safety net, {v})"

The safety net keeps a slow-but-good model visible when none of its runs crossed 70. Below 40, and empty diffs, rank on mechanics alone.

The blind spot is built in. The threshold tends to filter out the runs that misstate their work the most: Mistral-Small-4 had 0 of 6 runs above it, so no code review ever saw them. Honesty is therefore checked on every run. Inside a review it is a separate field that never changes overall; runs below the threshold get an honesty-only pass over CLAIMS.md. That pass, and what it found, is part 12 of the 128 GB box series.

What 75 reviews looked like#

Through the last review filed on 15 October: 75 reviews, no 5, three 4s, 69 with at least one blocker. The six without one are the three 4s and three of the 3s. Of the 34 runs whose verdict was PASS, 29 carry a blocker. By 12:26 on 16 October seven more had been filed, and all seven have one too.

71 of those 75 runs are the ledger task in its Java, Go and Zig versions, the other four a transfer service in Spring and Micronaut, all under two harnesses, and the selector had already removed the weakest code. "Most agent code is unmergeable" doesn't follow. What follows is narrower: a PASS from hidden tests says the code does what the task author tested, and the task author didn't cancel a context halfway through a transfer. The review score now sits next to the pass rate in every ranking from this study. That is how the stage-one ranking ended up putting the best-reviewed code on the slowest arm.

Filed under: ai-agents, code-review, testing, go, llm

Sources#

Review packets, rubric (RUBRIC.md), procedure (REVIEWER.md), prepare-review.sh, select-for-review.py and the review JSONs: private bench repository. Code excerpts are copied from those files; the Go excerpts come from the pilot run's final tree.
Probe c run: written and run on 2026-10-16 against a scratch copy of the pilot packet, Go 1.26.0. Counts in the last section computed from the committed review JSONs (75 files as of the 2026-10-15 20:21 commit, 82 as of 2026-10-16 12:26) and each packet's score.json.
context.WithoutCancel — Go standard library, since Go 1.21.
git add --intent-to-add — why untracked files appear in the diff.