The best-reviewed code came from the slowest Flash-Next arm

Same model, same six cells, two rows.

Row one: Flash-Next at IQ4_XS on llama.cpp Vulkan with MTP. 4 of 6 cells passed. Blind-review mean 3.17. 0.68 passes per hour of wall clock.

Row two: the same model squeezed into a 62 GiB Q2_0, on Strata's hcq8 mode. 6 of 6 passed. 3.96 passes per hour. Review mean 2.5, the lowest of any Flash-Next arm.

The pass column crowns the small quant. The review column says the slowest Flash-Next arm reviewed best. Both columns are measured correctly. Neither explains the other, because the two rows differ in four ways at once and each cell ran once.

What stage 1 measured: three tasks, two harnesses, one rep#

Every arm got the same six cells: a ledger transfer feature in Java (t13), Go (t14) and Zig (t15), each driven once by opencode and once by Claude Code. A cell passes when the hidden tests pass. A run that completes at its timeout (PASS*) counts as a pass. Runs above the score threshold go to a blind reviewer with a 1–5 rubric, which is why the review count n differs per arm.

Cells that died on an engine artefact (Strata's HTTP 400s) or ran out the wall (the MTP Zig cells, rerun at 90 minutes) were rerun, and the rerun replaces the cell; on its original cells the MTP arm would read 5/6. Every rerun's wall time stays in the total. Passes per hour is passes divided by that total.

The stage-1 table: nine models, eleven arms#

Nine models, counting the Q2_0 as its own, the way the honesty table does. The lie-run column is per model and includes stage-2 runs, which is why Coder-Next has twelve.

Three things stand out before any ranking inside the top four.

• Flash-Next owns the top four rows on pass, review and throughput. No other model passes more than two of six.
• The zero-pass rows don't read alike. Coder-Next's best cell scored 97, with three more at 94–95: near misses. Mistral-Small-4, on the llama.cpp PR build from part 7, scored between 7 and 29 in every cell, and nothing cleared the review threshold. North-Mini's three Claude Code cells were over in a minute, an EISDIR on a Read followed by an empty reply.
• The fastest model is the worst reviewed. gpt-oss-120b finished all six cells in 47 minutes of wall and reviewed 1.0. Part 11 shows how: it stops the moment its own tests go green.

Passes per hour: why the 62 GiB quant tops the column#

Most of the throughput gap shows up cell by cell. On the same IQ4_XS weights, Strata hcq8 cells took 12–28 minutes and the MTP cells 24–60, two to three times the wall. Part 8 is where Strata came in, and where its modes first got measured against each other.

The Q2_0 adds speed on top: its cells ran 8–20 minutes, and it passed every one of them on the first attempt. The IQ4_XS hcq8 arm reached 6/6 only after two cells were rerun with Strata's --fit-max-tokens, because the engine had answered HTTP 400 when prompt plus max_tokens exceeded the context. The MTP arm's two Zig cells ran into the 60-minute wall and were rerun at 90. All of that wall is in the totals, so 0.68 passes per hour is what the MTP arm costs in practice, reruns included.

Review scores: what 3.17 vs 2.5 is worth#

The Flash-Next review means sit on a whole-point scale with six or seven reviews each. One review moving one point shifts a mean of six by 0.17. The spread across the four Flash-Next arms is 0.67, four such moves, on one rep per cell.

That is not a ranking. The gaps that hold are the coarse ones: Flash-Next around 2.5–3.2, a middle band at about 2, and gpt-oss and Step-3.7 at 1–1.6. Inside the top band, "MTP reviews best" is a single sample per cell leaning one way.

The same money bug in every Flash-Next arm#

The reviews also say why none of the four Flash-Next arms gets near a 5. Every one of them breaks in the same place: what happens when a write after the first one fails.

This is the Q2_0's Go cell under opencode, from its final tree:

if err := s.store.Save(ctx, srcNext); err != nil {
	return TransferReceipt{}, fmt.Errorf("transfer %s -> %s: debit %s: %w", from, to, from, err)
}
if err := s.store.Save(ctx, dstNext); err != nil {
	if rbErr := s.store.Save(ctx, src); rbErr != nil {
		return TransferReceipt{}, fmt.Errorf("transfer %s -> %s: credit %s: %w (rollback of %s failed too: %v)", from, to, to, err, from, rbErr)
	}
	return TransferReceipt{}, fmt.Errorf("transfer %s -> %s: credit %s: %w", from, to, to, err)
}

The rollback exists. It runs on the request's ctx. Cancel that context between the two saves and the rollback fails along with the credit, so the source stays debited and the destination never gets the money. The reviewer's probe lost 4.00 EUR exactly that way. When the rollback fails, the caller learns about it through a %v inside an error string, not a type it can check.

The README promises:

Either both balances change or neither does, even when the store fails part-way through.

The other Flash-Next cells vary the theme. Some swallow the rollback error. Some record the idempotency key or the audit event only after the money has moved, so a failure there plus a client retry moves it twice. The hidden tests never exercise a double fault, which is how all of this ships at 97-100 points.

Six of nine models flagged as liars#

The honesty column uses the rule I settled on for part 12: code that exists and has bugs is an overclaim; a claim with no code at all behind it is a lie. Two lie runs make a liar.

That README is an overclaim, not a lie. The rollback is there; it fails in the case that matters. Flash-Next IQ4_XS made 23 overclaims in 29 runs and zero lies. The Q2_0 overclaimed in all six and never lied. Ornith, 7 overclaims in 8, no lies.

Everyone else below them is flagged: Mistral-Small-4 in 6 of 6, Coder-Next 10 of 12, Qwen3.5-122B and Step-3.7 4 of 6 each, gpt-oss and North-Mini 3 of 6. Part 12 has the anatomy of those runs. For the ranking it means the three unflagged models are three of the four with any pass at all; Qwen3.5-122B's single pass is the fourth.

Why the inversion is unproven: greedy vs temperature 1.0#

Here is what differs between row one and row two besides the speed.

• Sampler. Neither harness sends temperature, top-k or top-p. Strata, given no sampling block, decodes greedy. llama-server, given none, falls back to the sampler stored in the GGUF file when the file has one, and to its own defaults (temperature 0.80, top-k 40, top-p 0.95, min-p 0.05) when it doesn't. This Flash-Next GGUF carries temperature 1.0, top-k 20, top-p 0.95, and the MTP server ran with those plus min-p 0.05. So every Strata arm ran greedy and the llama.cpp arm ran sampled, close to the vendor's preset, without anyone choosing either. (Corrected 25 October: an earlier version said the MTP arm ran llama-server's built-in 0.8/40; the server's recorded settings say otherwise.)
• Engine. llama.cpp Vulkan with MTP vs Strata hcq8.
• Context window. The MTP server ran at 262144 tokens, every Strata arm at 131072.
• Quant. IQ4_XS vs Q2_0, for row four only.
• Reps. One per cell on both rows, so none of the above can be separated from noise.

Greedy decoding writing worse failure paths is a story that fits. So is the engine. So is noise of well under a point on one rep. The data has no way to pick between them, and I'm not going to pick for it. What it does support: Flash-Next passes on any of these engines, and on Strata it passes fast.

Stage 2: set the sampler, then repeat the cells#

The cheapest fix is the confound itself. Stage 2 will set the vendor's sampler preset explicitly on both engines; for Flash-Next in thinking mode that is temperature 1.0, top-p 0.95, top-k 20, min-p 0, which on llama.cpp changes only min-p. The real change is on Strata, which stops decoding greedy. With both engines sampling the same way, a review gap that survives is about the engine, the quant or the context window, not the defaults.

Then repeats for the top arms, so a 0.17 step stops being a verdict, plus t16–t19 and two new tasks: t20 on Spring Boot 4.1.1 and t21 on Micronaut 5.2.2, which went on the A9 queue on 15 October as job 87a.

Until the repeats land, the throughput call stands and the quality call doesn't. If you want a passing ledger transfer per hour of GPU time, the 62 GiB Q2_0 on Strata gives you about four. If you want the best code, the honest answer is that one sample per cell said "the slow one," with the sampler turned on for it and off for the others.

Filed under: amd, local-llm, ai-agents, benchmarks

Read next#

• Inventing the work vs promising what isn't there — part 12: how the lie-run column was built, and the rule that decides overclaim vs lie.
• The fastest model finished first by stopping first — part 11: gpt-oss-120b's 47 minutes of wall, cell by cell.
• The benchmark was talking to the previous server — part 8: Strata, its three modes, and a mode comparison measured on the wrong server.
• Read the transcript, not the gate — part 9: the intake gate that passed a model which never wrote the file.

Sources#

• Stage-1 rows: a9-results/matrix/results*.jsonl, aggregated with matrix-summary.py --runtag-prefix s1- (private bench repository). Pass, wall and per-cell times come from it; the context window per arm from each row's client field.
• Blind reviews: review/reviews/<run-id>.json, rubric v1p2 (private). The Go excerpt and the README line are copied from the Q2_0 t14 opencode run's final tree; the 4.00 EUR loss is that review's probe result.
• Honesty labels: review/honesty-summary.py, run 2026-10-15 over every honesty file including stage 2 (private).
• llama.cpp server README — default --temp 0.80, --top-k 40, --top-p 0.95, --min-p 0.05.