testing
- The fixes had the bugs
Seven re-review passes over a Go LLM proxy found 11, 8, 10, 4, 2, 1 and 0 bugs, nearly all in the fixes. Two could have crashed the process.
- Pulling plugs on purpose
The first power-bank test passed because a second supply was hiding the failure. What each test in the UPS project could prove, and what it couldn't.
- No Manual Intervention Required
An LTS-to-LTS Quarkus bump, 3.33 to 3.40, changed no production Java and broke the tests twice: one class loaded twice, one database shared by six branches.
- Health 200, API 200, Prometheus 000
A GraalVM 25i4 bump made native /prometheus stop answering. The reflection gap was old; a new default turned a missing metric into a hang.
- How to tell when your coding agent lied, not just failed
Grade what a coding agent says, not only its code: buggy code is a bug, promised code that doesn't exist is a lie. The script, the rule, the checker's traps.
- 29 of 34 passing agent runs had a blocker the hidden tests never saw
Blind review packets, a merge rubric and probes that must run: 34 agent-written money transfers passed the hidden tests, and review found a blocker in 29.
- Inventing the work vs promising what isn't there
Checking 71 local-model agent runs' claims against their own command logs: three models invent work, three promise atomicity they never built.
- The fastest model finished first by stopping first
gpt-oss-120b cleared six agent cells in 51 minutes by quitting early on each. Two of those stops shipped Zig that does not compile.
- Read the transcript, not the gate
15 of 17 local models passed the agent smoke gate on the 128 GB box. Eight of those passes hid a Claude Code failure the intake had logged and nobody read.
- The deadlock test can't see a half-held lock
Every model wrote a thread test. Most of them could not fail: with locks taken in id order, opposite-direction transfers serialize on one mutex.