Empty diff, finished code: 124,000 characters of thinking

Assumes you've run a coding agent against a local llama.cpp server and know what reasoning tokens are.

16 October, the A9, K2-Horizon Q8 on llama.cpp master Vulkan, Claude Code on a Java ledger ticket:

• 19:48:50 the session starts.
• 19:51:45 the last of 14 file reads comes back. By then the model has read the README, eleven main classes and two test classes.
• 20:34:44 the next turn ends. It is one block of thinking, 140,241 characters long, and it stops mid-sentence.
• 20:48:50 the 60-minute timeout kills the process.

Files changed: 0. Score: 29 points, the score the matrix gives an untouched repository. The obvious reading is a model that sat in a loop for an hour. The 43-minute turn in the middle says otherwise.

Empty diff, 29 points, two harnesses#

The same model left a second empty diff that night. OpenCode, the Go port of the same ticket, started at 20:48:59, right after the Java cell:

The Go cell ended by itself with nothing written, about 21 minutes before the limit (3,600 s minus 2,348 s, my subtraction).

It is not the context window either. Part 14 was about that wall, so I checked it first: the context-cap grep is 0 on all six K2 cells. These two were served at 262,144 tokens, and the context in them peaked at about 62k and 75k.

One turn, 32,000 tokens, no tool call#

Both long turns stop at the same number. The OpenCode export records finish: length with 32,000 output tokens. The Claude Code transcript records stop_reason: max_tokens, also 32,000. llama-server logged both requests, Java first, then Go:

106.43.183.674 I slot print_timing: id  0 | task 45137 |        eval time = 2570684.43 ms / 32000 tokens (   80.34 ms per token,    12.45 tokens per second)
160.05.034.707 I slot print_timing: id  0 | task 90248 |        eval time = 2224622.30 ms / 32000 tokens (   69.52 ms per token,    14.38 tokens per second)

32,000 is the output limit both harnesses ask for on every request. The Zig context post caught OpenCode sending exactly that in a payload capture. At 12 to 14 tokens per second, filling it took 43 minutes in Java and 37 in Go. Both figures are my arithmetic from the timestamps.

Neither turn contains a single tool call. The cap ended each turn while the model was still in its reasoning. The tool call would have come after the reasoning.

What 124,000 characters of reasoning contain#

The Go turn is 123,942 characters, all reasoning, no visible text. The structure is easy to follow:

• Requirements and design notes: where to keep idempotency records, which lock to take first, what rollback means, where the overflow check goes.
• func (s *Service) Transfer(ctx context.Context, from, to AccountID, amount Money, idempotencyKey string) (TransferReceipt, error), drafted twice.
• A README section, "Transfers", with a Mermaid sequence diagram.
• Under a transfer.go heading, the final file: 117 lines, receipt type and helpers included.
• The small edits elsewhere: three new sentinel errors, one audit event type, one statement case.
• Then the tests. Just before the end it catches a sign error in one of its own expected balances and redoes the sum. The cut lands inside a test function, in the middle of an if.

The Java turn has the same shape at 140,241 characters. transfer() is drafted eight times, there are 19 @Test methods, and the cut lands mid-sentence in the README's concurrency paragraph.

K2 was not stuck. It was working through the whole ticket, in order, inside a channel that never reaches the working tree. The harness post measured how much a client makes a model think. This is what happens when that thinking reaches the ceiling.

Does the code in the reasoning work?#

To find out, I copied the drafted blocks out of the Go turn into the archived starting tree, verbatim apart from indentation where a snippet was spliced into an existing file. The new transfer.go, the struct fields and constructor line, the sentinels, the event type, the statement case and the README section went in. No tests went in, because the reasoning never finished one. Then I ran the cell's own scorer on it in a golang:1.26.0 container, the A9's Go version. As a control, the untouched tree scores 29 and fails all 15 hidden core tests, the same as the matrix row.

The rebuilt tree, the first 17 lines of the scorer's output (two NOTE= lines follow):

VERDICT=PARTIAL
TESTS=none
METRIC json_impl=na-ledger
METRIC race=on
METRIC files_added=1
METRIC files_changed=5
METRIC files_deleted=0
METRIC own_tests=0
METRIC own_tests_failing=0
METRIC own_concurrency_test=no
METRIC hidden_core=15of15
METRIC hidden_detail=12of12
METRIC mutants=na-notests
METRIC mermaid=ok
METRIC new_sentinels=ErrIdempotencyConflict+ErrIdempotencyKeyRequired+ErrSameAcco
METRIC transfer_sig=ctx-keylast
METRIC points=80

go vet and gofmt are clean, the existing tests pass, and all 27 hidden tests pass under the race detector. The 20 missing points are exactly the "own tests" and "mutation" rows: the tests it was writing when the cap hit.

The code is not perfect. The rollback after a failed second save runs on the request's context. I wrote a probe that cancels that context right after the debit is saved:

    zz_probe_cancel_test.go:39: err=transfer probe-key: context canceled A=7.00 EUR B=0.00 EUR

Account A started with 10.00 and sent 3.00. B got nothing, and the 3.00 is gone. It is the same bug as in the one K2 cell that passed, where the review probe left A at 700 and B at 0 minor units, and the same one the stage-1 ranking found in other arms. On this probe the empty diff held code no worse than K2's best submitted run, and that run scored 100.

Two harnesses, two ways to lose the work#

What happened after the cap is where the harnesses differ.

OpenCode stopped. The export has nothing after the length turn, the session closes, and the runner records a normal exit. Nobody asked the model to continue.

Claude Code tried to recover. It added a message telling the model that it had hit the output limit, to carry on from where it was cut off, and to break the rest into smaller steps. The model started generating again. But the retry's prompt held about 42,600 tokens: the conversation from before the long turn, without the 32,000-token thought. That is my inference from the server's numbers. The model was asked to pick up a thought it could no longer see. About 11,700 tokens into the retry, the hour ran out:

120.54.313.243 W srv          stop: cancel task, id_task = 77138
120.54.389.556 I slot      release: id  0 | task 77138 | stop processing: n_tokens = 54341, truncated = 0

That recovery does work when there is time for it. ISTA Coder hit the same cap on the Zig ledger task under Claude Code on 15 October. On Strata at 68.5 tok/s, its 32,000 tokens took 471 seconds. Its cut turn was itself a loop (more on those below). Claude Code's retry still ended in a tool call, about 50 more turns followed, and the cell passed with 100 points. K2 reached the cap at minute 46 of 60 (19:48:50 to 20:34:44).

None of this is a model lying about its work. Part 9 had a model that claimed a file it never wrote. K2 never claimed anything in these two cells.

K2 plans in one turn, usually under the cap#

K2's habit of planning everything in one turn shows up in all six cells. Only two crossed the line:

In the other four cells, a turn of 13,000 to 27,000 tokens ended in a tool call and the work landed on disk. The habit itself is fine. The trouble is the margin: one of the turns that fit was at 84 % of the cap (26,920 of 32,000).

Same score, opposite diagnosis#

K2 is not the only model this happened to. I counted the 153 runs archived on the A9 through the GLM-5.3 arm on 17 October. In 15 of them a turn ends on the cap, and in 13 the cut turn has no tool call. Nine of those 13 are empty diffs. All nine score 29 points, or 9 on the C task, which gives less for doing nothing.

The nine do not all fail the same way. My first sort used two crude measures from the cut turn: characters per token, and the share of distinct lines. GLM-5.3 broke both, so a third joined them: the share of unique word 8-grams in the turn's last 3,000 words.

Four of the nine are loops, the failure the score suggests. GLM-5.3's Claude Code turn reasons for under 7,000 characters, then writes AAAA. until the cap. Its OpenCode turn spends its last 43,000 characters on "Multiplying 8 by 200 = ~ 400×8×1." Each loop runs inside one line (the longest lines are 58,457 and 51,165 characters), so the repeats never show up as repeated lines, and the distinct-line share scores the two turns 1.00 and 0.94, above every draft. Their 2.04 and 2.72 chars/token sit well above Qwen's 1.06.

The 8-gram share is what splits them: 93 to 100 % for the drafts, 0 % for GLM, 33 % for Flash-Next re-arguing the same concurrency case. It has its own blind spot. Qwen3-30B's 00-00-00-… tail has no spaces, so to a word counter the whole 33,936-character turn is 391 words, and the share reads 100 %. Only chars/token catches that one: 1.06.

So it takes two measures, one per kind of loop, and both are still heuristics. The matrix gives all nine the same 29, and that number hides the one difference that matters.

The counter that was switched off#

The bench already knew about this. report-sweep.py counts output-cap hits in the bench's own serve log. Its docstring says why: a cap hit scores FAIL and "looks exactly like a wrong answer". The case that prompted it was an 8192-token cap on 22 September.

Every A9 arm runs against a server the bench did not start. For those runs run-scenario.sh added a note to the result row instead of counting. Every K2 row carries it:

no bench serve log, output-cap hits not counted

Yet the archived serve.log has the exact eval time = … / 32000 tokens line the counter greps for. The evidence was saved, but the counter never looked there.

As of 17 October, review/cap-hits.py counts from the archived sessions instead, which every run has. For the Go cell:

20261016T212812-k2-horizon-q8-opencode-t14-ledger-go-base-s1-vknewr1: output-cap hits 1 (reasoning-only 1); ctx-full length stops 0; serve.log 32000-token turns: 1

The rerun at 65,536 tokens#

The rebuild still needed my hands. So I ran the five drafted cells again: same model, llama.cpp build, task and harness, with one change, the per-reply output cap raised to 65,536. The two Ornith cells on t16 and t17 got a 90-minute wall, where the originals had 60. All five came back with code that passes the hidden core tests, or all but one of them in C:

None of them hit the new cap. For the K2 Go cell, the one I rebuilt by hand:

20261017T122304-k2-horizon-q8-opencode-t14-ledger-go-base-s1-c64vknewr1: output-cap hits 0 (reasoning-only 0); ctx-full length stops 0; serve.log 65536-token turns: 0

That K2 rerun wrote the code itself: 30 tool calls, 11 tests of its own, 7 of 7 mutants caught, all 27 hidden tests green. The hour ran out after the work was on disk, so the row says FAIL and the artifact PASS, 100. It kept the bug, too. A cancel probe started the two accounts at 10.00 and 5.00 and ended them at 9.00 and 5.00. The 1.00 in flight is gone.

The column I did not expect is the largest reply. In four of the five reruns no reply reached even the old cap: 20,729 to 27,682 tokens. Only Ornith's TypeScript cell wrote one the old cap would have cut, 57,757 tokens; the other four planned in shorter turns this time. Ornith had already passed the same Go cell once, in its pilot run on 12 October, under the old cap. For that cell, the 32,000-token cut was one bad draw.

The higher cap buys margin, and K2 needs it. Its Go rerun still planned in one reply of 27,682 tokens, 86.5 % of the old cap (my division).

What changed in the bench#

The cap is 65,536 for every A9 run started after the probe, since 12:44 on 17 October (run-scenario.sh, 5fae52f). Earlier rows, the GLM-5.3 arm that morning included, ran at 32,000, and BENCH_OUT_CAP=32000 reproduces them. The wall changed the same day: every task now gets 4 hours (a9-arm.sh, fb11355), and each row's tmo_s records the wall it had.

With a 4-hour wall a loop could hold the GPU for all of it, and the bigger cap gives it twice the room per reply. So review/loop-guard.py (bdb207e) now runs beside every Claude Code and OpenCode cell. It scores each capped reply with no tool call on its last 3,000 words. Under 60 % unique 8-grams or 1.5 chars/token counts as a loop, and two loop replies stop the cell. Run offline over the 13 archived reasoning-only cap hits, it flags every loop and passes every draft. Each of those cells had one capped reply, so none would have been stopped, ISTA's recovery included.

A 29 with a cap hit next to it still means: open the cut turn. Even a loop there does not prove nothing landed: GLM-5.3's Claude Code Zig cell looped at the cap and still scored 70 from files it wrote in other turns.

The rebuilt Go cell stays out of the ranking. Its 80 came from code that only ever existed in the reasoning. The rerun's 100 needed no rebuilding: K2 wrote that code to disk.

Filed under: local-llm, ai-agents, benchmarks, opencode, claude-code

Read next#

• Seven FAILs, one setting: the other wall that looks like a FAIL, the context window instead of the output cap.
• Read the transcript, not the gate: the opposite case, a model that claimed a file it never wrote.
• The harness decided how long the model thought: how much thinking each client draws out of the same model, below the ceiling.
• The best code came from the slowest arm: the rollback-on-cancelled-context bug this code shares with K2's one PASS.

Sources#

All primary files from my own bench; none are public.

• K2-Horizon session exports: A9 ~/bench-archive/20261016T204858-k2-horizon-q8-claude-t13-ledger-base-s1-vknewr1/ and …/20261016T212812-k2-horizon-q8-opencode-t14-ledger-go-base-s1-vknewr1/ (sessions/, serve.log, work.tar.zst).
• ISTA Coder Claude Code t15: ~/bench-archive/20261015T200119-ista-coder-iq1m-claude-t15-ledger-zig-base-s1-istahcq8r1/.
• Matrix rows: a9-agent-bench agent-playground/bench/a9-results/matrix/results-s1-vknewr1.jsonl, results-claude-s1-vknewr1.jsonl.
• Scorer and hidden tests: agent-playground/bench/scorers/t14.py, scorers/hidden/t14/ (a9-agent-bench ea44303).
• Cap counting: agent-playground/bench/report-sweep.py (docstring), run-scenario.sh (external-server note), review/cap-hits.py.
• GLM-5.3 REAP cut-off cells: ~/bench-archive/20261017T032358-glm53-reap-q4km-claude-t13-ledger-base-s1-vkr1/, …/20261017T061156-glm53-reap-q4km-opencode-t15-ledger-zig-base-s1-vkr1/; all cap hits: review/cap-hits.jsonl (153 runs); 8-gram test: review/loop-test.py.
• Cap-probe reruns: matrix rows results-s1-c64vknewr1.jsonl, results-claude-s1-c64vknewr1.jsonl, results-s1-c64vkr1.jsonl, results-v7bc64r1.jsonl; archives ~/bench-archive/20261017T*-*c64*; reviews review/reviews/20261017T*c64*.json.
• Bench changes: run-scenario.sh default cap 65,536 (5fae52f), review/loop-guard.py (bdb207e).