Zig needed context, not time

Assumes you've driven a coding agent against a local llama.cpp server and know what context compaction is.

When an agent run dies at the wall clock, the obvious diagnosis is that it was slow, and the obvious fix is a longer wall. Four 27B models on a Zig task were hitting a one-hour limit in about half their runs at 64k context, so twelve runs of the test-first prompt got four hours instead.

They produced zero passes. The same models at 128k context, with the same four-hour limit, passed all 15 hidden core tests in 36 runs out of 36, and 27 of those 36 finished inside the old hour. One of the 36 later read the hidden tests, so 35 of those passes are clean; it had cleared all 15 before it could.

The extra hours got the 64k runs further. They did not get them a pass. One caveat I found later: the four-hour runs also started with about 8k more prompt tokens than the one-hour ones, so they got more time and less room, not more time alone (details below).

The setup: one Zig task, four models, three prompts#

The task is t15-ledger-zig from a local agent benchmark: add an idempotent, atomic, thread-safe transfer to a small Zig 0.16.0 ledger service. A scorer then runs 15 hidden core tests and some detail checks (docs, a Mermaid diagram, extra validation). Any failed core test makes the run a FAIL, and so does hitting the time limit, whatever the code left behind. Detail misses alone make it PARTIAL.

The models are four IQ4_XS builds of Qwen3.8-27B, served by llama.cpp on an RTX 3090: the stock model (qwen38) and three fine-tunes of it (swift15, thinkingcap, qwen38pi). The client is OpenCode 1.18.4, reasoning effort medium, each model's own card sampler on the server. (Found later: OpenCode 1.18.4 overrode top_p to 1 for ids containing "qwen", so qwen38 and qwen38pi sampled at 1.) Three prompt styles:

• spec: the task, with a bullet asking for tests including a threaded one.
• TDD: the same, with that bullet swapped for test-first plus a mandated stress test.
• review: the spec run, then a second turn listing the names of the scorer checks it failed.

Each cell is three runs per model, twelve runs total. Keep that in mind for everything below.

Seven cells, one table#

At 64k and one hour, every prompt style lands at one or two passes in twelve. The stock qwen38 never passed a single hidden core test in any of its nine 64k one-hour runs.

The control: four hours at 64k, zero passes#

The 128k arm changed two things at once, the window and the time limit. The control changes one of them back (it also ended up carrying a larger prompt, covered below): the TDD prompt at 64k, with the 128k arm's four hours.

Time helped. Seven of twelve runs reached all 15 core tests, against three in the one-hour TDD cell. Nobody passed. The one-hour cell had a single PASS, so at this sample size zero reads as noise. It does not read as progress.

Given four hours, no 64k run stopped inside the first one. The fastest took 3,914 s. They spent the time compacting: between 22 and 58 compactions per run, median 31. At 128k the median was 1.

qwen38 shows where the extra hours went. Two of its three control runs rode the 14,400-second wall to the end, after 58 and 55 compactions, with no core test passing. One of them ended four hours of work with test code that doesn't compile. From its score sheet:

tests/transfer_test.zig:24:30: error: no field or member function named 'transfer' in 'service.Service'

Its tests call a transfer method that its Service never got.

128k: every run through the core tests, 35 cleanly#

Same models, same prompts, 131,072 tokens of context, four hours. Every one of the 36 runs passed all 15 core tests, 35 of them cleanly; the 36th is a review run that read the hidden tests, covered further down. The clean verdicts were PASS 4, 6 and 8 of 12 for spec, TDD and review, with the rest PARTIAL apart from that one run. No run in any 128k cell failed.

Most didn't need the extra time. 27 of the 36 finished within 3,600 seconds: 11 spec, 8 TDD, 8 review (a review run's wall covers both turns). The slowest, qwen38 on TDD, took 7,985 s. Six of the nine runs over the hour were qwen38's.

The hardware did not favour 128k either. Serve logs show the 128k arms decoding at 27.2 to 28.8 tokens per second against 34.0 to 36.2 at 64k, since decode slows as context deepens. The GPU's power cap dropped from 320 W to 280 W on 9 and 10 October, during some 64k one-hour runs; every 128k run was at 280 W. Any speed advantage went to 64k.

What "64k" meant in OpenCode: 33.5k to 35k#

The bench's per-run OpenCode config sets limit.context to the server's context and limit.output to 32,768. A payload capture shows OpenCode sending max_tokens 32,000. OpenCode stores token counts for every request in its session database. The last request before each compaction shows how full the window was when it fired, and it was never near 65k:

• 64k cells: median prompt at compaction 35.0k to 35.5k per cell. The lowest of all of them was 33,541. 65,536 − 32,000 = 33,536.
• 128k cells: medians 99.6k to 100.0k, lowest 99,096. 131,072 − 32,000 = 99,072.

So the working window looks like context minus the output budget. This is an inference from those numbers. I have not checked it against OpenCode's source, nor how its documented compaction.reserved buffer fits in. Either way, the 64k runs compacted between 33.5k and 35k, and 128k gave them roughly 2.8 times the room.

That window is smaller than what the models read before writing. Of the 48 runs at 64k, across both time limits, one wrote or edited a file before its first compaction. At 128k all 36 did.

Compaction contributed. It wasn't the whole story#

The clearest victim is qwen38. One of its 64k spec runs made 123 tool calls, compacted 8 times and edited nothing. Each of its eight summaries recorded that no code had been written yet. Another run made 153 calls and one edit, and 7 of its 9 summaries said the same. After each one it re-read the same source files, ran out of room again, and summarized again.

One review run lost the task itself. Its third compaction produced a "summary" of 1,163 characters of raw tool-call markup with no objective in it. In its next turn the model asked itself what the task was and noted it couldn't find a task file. It then rebuilt the task from run-1.log, the harness's live log of its own session, which happened to sit in the workspace. It made 138 tool calls without one write.

That does not make compaction the sole cause, and the counter-evidence comes from a sibling tune. qwen38pi compacted about as often (29, 40 and 45 times in its control runs) and still wrote the code: two of those three reached all 15 core tests. qwen38's own 128k runs also re-read files after compacting and still passed every core test, because by then the code existed. No arm ran 64k with compaction turned off, so "caused" stays unproven. The fair word is contributed: strongly for qwen38, survivably for the others.

The prompt I didn't write: my ~/.claude in every run#

On 14 October a payload capture against a fake server found something else in that window. OpenCode's Claude Code compatibility, on unless you turn it off, had put my global ~/.claude/CLAUDE.md and 22 of my Claude skills into the system prompt of every bench OpenCode run.

Of the 53,483, 28,504 characters were the CLAUDE.md and 15,901 the skill list. With the flag, three skills remained, about 1.7k characters. OpenCode's session database has the real count, in tokens, for the first request of every run: system prompt, tool definitions and the task. During the control it came to about 20.3k. Runs with the flag set, from 15 October on, come to 8.4k to 8.6k, so close to 12k tokens of it were mine.

It also grew mid-benchmark, and its git history can't say when. Those commits were made on my laptop; the bench box only became a clone of that repo on 11 October at 15:56 and pulled again on 13 October at 17:28. The per-run token counts can. The first request was 11.9k to 12.1k tokens in the one-hour spec and TDD cells, 19.0k to 19.2k after the clone, and 20.3k after the pull. The one-hour review cell was split between 12.0k and 19.0k. The 128k cells sat at 19.0k to 19.2k, apart from three TDD runs at 12.1k. The control, the arm meant to isolate time, ran last: 19.2k for its first run and about 20.3k after that, against 12.1k in the one-hour TDD cell it is compared with. That is about 8k more tokens before the first file read. With compaction at 33.5k, the control had roughly 13k tokens of room after its first request, and the one-hour TDD cell about 21k.

What this does and doesn't change:

• The control is not the clean time-only test it was meant to be. It had nearly 40% less room after its first request than the one-hour TDD cell, so its zero passes mix more time with less room. No rerun has measured what that cost; settling it would mean running the four-hour 64k arm again with the flag set. The prompt's growth can't be what separates 64k from 128k, though: the one-hour 64k cells mostly carried the smaller one, and the 128k cells carried nearly the control's and still had about 80k of room.
• The 64k numbers are pessimistic. During the control, the first request filled about 60% of the 33.5k window before the model read a single file; in the 128k cells it filled about 19% of 99k. Without my ~/.claude it would be about a quarter and under a tenth. Some of the gap between 64k and 128k is this tax rather than the setting itself.
• pi rows never had it, so OpenCode and pi results from this period are not comparable on context.

None of this is OpenCode misbehaving: its docs describe the fallback, and say a ~/.config/opencode/AGENTS.md takes the place of ~/.claude/CLAUDE.md. The bench now sets the flag for every run. If you run OpenCode with a large ~/.claude of your own, set OPENCODE_DISABLE_CLAUDE_CODE=1. The rules docs also list OPENCODE_DISABLE_CLAUDE_CODE_PROMPT and OPENCODE_DISABLE_CLAUDE_CODE_SKILLS for disabling one half. Only the full flag was measured here.

Bench bugs that didn't move these verdicts#

Three things in the Zig rows look like failures and aren't:

• baseline=FAIL on every early row. zig wasn't on PATH in the systemd unit when the pristine fixture was built, so every Zig row before 9 October 21:10 recorded a failed baseline. The scorer finds zig itself, so verdicts were unaffected; the rows were corrected within hours.
• A PASS with zero tool calls. The harness counts tool calls by grepping run-1.log. Two 128k PASS rows no longer have that file and record 0; OpenCode's database has 116 for the swift15 run and 129 for the thinkingcap one. Both runs deleted the log themselves, and the thinkingcap one is the tainted run in the next section.
• 0of0 hidden core means the tree didn't compile. No test ran at all.

Two harness leaks are real confounds rather than bookkeeping, and a third gets its own section below. run-1.log sat inside the workspace, and models read their own live log as a previous agent's run. Two control runs of qwen38pi also read notes in the repo root. These two are fixed now, and every row above predates the fixes.

One review PASS read the answer key (16 October)#

thinkingcap's first 128k review run is tainted. The harness scored turn 1 into a turn1/ directory beside the workspace and left the scorer's work copy there, hidden test suite included. Turn 1 had failed one detail check, a memory leak. In turn 2 the model listed the directories above its workspace, ran find on the sibling .meta directory, and grepped the suite's helper file, zz_hidden_helpers.zig, three times, the first time for the memory scenario it had failed. Near the end it ran rm run-1.log run-2.log. The answer-key scan reads only those logs, so the row carries no flag. Its verdict went from PARTIAL to PASS.

That PASS doesn't count. Review at 128k is 8 clean passes in 12, and the three 128k cells hold 18 clean passes. I keep its core result with an asterisk: the turn-1 score sheet, written at the end of turn 1, before the model could read the copy, already has all 15 core tests passing. The bench has kept that copy out of the model's reach since 16 October.

Since then: no MTP, the quant, and the docs checks (16 and 22 October)#

Three results came in after these runs. None of them reverses the 64k-versus-128k comparison.

• Every row above ran llama.cpp without MTP. The bench's serve script only gained the multi-token-prediction flags on 15 October. Where a task ran both ways, MTP roughly halved the wall: 169 s against 90 s per cell on three short tasks, and 19,755 s against 10,309 s for swift15's nine 128k Zig runs. The control still got four times the one-hour arm's time budget on the same engine, so its zero still counts. The swift15 rerun, which also dropped the ~/.claude prompt, passed 3 of 9 against the original 6, with all 15 core tests in 8 of 9. Three runs per prompt can't separate either change from noise.
• The quant moved PASSes more than core tests. On a Q4_K_M build, swift15 passed 7 of 9 both on llama.cpp with the rerun's settings and on a second engine, open-jet, against 3 of 9 for IQ4_XS in the rerun. All three arms cleared the 15 core tests in 8 of 9 runs; IQ4_XS lost mostly on one memory-leak detail check. The table's own IQ4_XS swift15 runs passed 6 of 9, so this is a lead, and the core-test column stays the sturdier claim.
• The scorer stopped grading docs. Since 22 October, the README checks (a Transfers section naming idempotency, failure and lock order, a Mermaid diagram) and zig fmt are recorded but no longer move a verdict. Rescoring the saved workspaces moved five rows here from PARTIAL to PASS: two qwen38pi runs (one of them a run that read notes in the repo root) and one thinkingcap run in the control, and one qwen38 run each in the 128k spec and TDD cells. The control is now 3 / 4 / 5 against 5 / 7 / 0 and 7 / 5 / 0 at 128k. No core test moved, and the 128k cells now hold 20 clean passes.

The missing arm, and the small n#

Three runs per model per cell is enough to see 0 PASSes in the control against 18 clean of 36 at 128k (3 against 20 under the 22 October scorer, or 2 without the control run that read notes in the repo root), and too few to rank prompts: spec's 4 PASS (5 now) against review's 8 clean at 128k is worth chasing and no more. Review is also scorer-guided, since its second turn names the failing checks. And even setting its larger prompt aside, the control isolates time in one direction only. Nobody ran 128k with a one-hour limit, but 15 of the 18 clean PASSes at 128k finished inside the hour anyway (16 of 20 after the rescore).

At 64k the agent spent its hours rereading. At 128k it spent them on writing code.

Filed under: local-llm, llm, ai-agents, benchmarks, zig

Read next#

• I blamed OpenCode; a 2-bit model passed Zig in it — the same Zig task at 64k, and the experiment that cleared the client as the cause.
• The deadlock test can't see a half-held lock — what the 36 core-clearing 128k runs' own concurrency tests could and couldn't catch.
• Zig's defer ends at the closing brace — the one-line Zig bug behind a 97-point FAIL in the 64k spec cell.

Sources#

• OpenCode docs: Rules, Claude Code Compatibility — ~/.claude/CLAUDE.md and ~/.claude/skills/ as fallbacks, and the OPENCODE_DISABLE_CLAUDE_CODE* variables.
• OpenCode docs: Config, Compaction — compaction.auto: "Automatically compact the session when context is full (default: true)".
• Benchmark artifacts (local, not published): the re-eval report tables (groups A and B, final 2026-10-15), per-run score sheets and results rows, the OpenCode session database (read-only: compactions, prompt sizes, tool calls, summaries), the 2026-10-14 client payload capture, serve logs and gpu-watch samples.