Seven FAILs, one setting: rerunning agents at 262144 tokens

A FAIL in the agent matrix reads like a verdict on the code. The agent wrote something, the hidden tests caught it, next row.

Three of the FAIL rows from the A9 score 100 points. Every hidden test passes in them, and every mutant is caught. The code is finished and correct. What failed was the session. Each one had grown to just under 131072 tokens of context when the next request crossed the line. The server refused it, and Claude Code exited with an error. The matrix grades that exit as well as the code.

Four more FAILs went the same way, with real gaps in the code at the moment of the cut. Seven cells, one changed setting: serve them at 262144 and run them again. Six passed. The seventh failed on its own code, so for the first time one of these seven verdicts is about what the agent wrote.

What a FAIL row records: the exit, not only the score#

Each cell is one task driven once by one harness: a ledger transfer in Java (t13), Go (t14) or Zig (t15) from stage 1, or the Spring Boot (t20) and Micronaut (t21) transfers from stage 2. A scorer runs hidden tests and mutants and produces points. The row also records how the harness exited. All seven rows below carry status prompt1-rc1: Claude Code returned exit code 1. A harness that exits with an error is a FAIL, whatever the points say. The one exception is a wall-clock kill whose code already passes, which counts as PASS*.

These are the seven cells as they were at 131072:

They don't fail alike. The top three are the cap and nothing else. The Flash-Next pair passed every core test but lost points on two detail tests and on unfinished work: when the session ended, t20 had no README section yet and t21 had no tests, docs or commit. Laguna-XS was failing two core tests in both cells when it was cut off. A cut run can't say whether it would have fixed them.

Strata and llama-server refuse at 131072 instead of truncating#

Both engines in these rows behave the same way at the wall: they refuse the request. Strata, which serves the Flash-Next and Swift arms, says so in its serve log. llama-server, which served Laguna-XS, says it differently. These are the first refusals in the Swift-1.5 t13 and Laguna-XS t14 archives:

[strata] 400 invalid request: request exceeds the context window: prompt (131115 tokens) leaves no room to answer in the context (131072); requests are never truncated
84.02.330.687 E srv    send_error: task id = 130522, error: request (132149 tokens) exceeds the available context size (131072 tokens), try increasing it

"Requests are never truncated" is the honest design. An engine that silently drops the start of a 131k prompt would hand the agent a session with the task description missing. Strata won't do that, so the request dies instead. Claude Code relays the 400 as an API error and stops. In the Swift-1.5 t13 transcript the prompt climbs from about 30k tokens on the first request to 131,006 on the last good one, with no compaction anywhere on the way.

An earlier fix took care of a different wall. Claude Code asks for max_tokens 32000 on every request, and Strata used to refuse whenever prompt plus 32000 exceeded the window. Three stage-1 cells died that way at just over 99k prompt tokens. Since 13 October the serve script starts Strata with --fit-max-tokens, which shrinks the answer budget to fit. Part 13 counts those reruns. What's left after that fix is the prompt alone outgrowing the window, and no flag fixes that.

Why 131072: the floor became the ceiling#

Part 5 set 128k as the minimum context for agent work on this box. The Strata configs written at setup took that minimum literally: --max-context 131072, for every Strata arm. My model roster gave Laguna-XS the same 131072. The floor was left in place as the limit.

The models allow more. Flash-Next is native 256k. Laguna-XS says so in its own file. A short script that reads the GGUF header, filtered to the context keys, prints for the Q8_0 on the A9:

laguna.context_length 262144
laguna.rope.scaling.original_context_length 8192

That is 8192 trained, stretched 32 times with YaRN to 262144. My serving config gave it 131072, and two of its Claude Code cells ran into exactly that. Reading the trained context out of a GGUF is its own manual.

Twelve archives carry the error; seven were left to rerun#

Every run on the A9 is archived with a slice of the serve log. One grep finds the runs that hit a context wall on either engine:

command grep -l -E 'exceeds the context window|context size \(' ~/bench-archive/*/serve.log

It lists 12 archives. Three are the stage-1 Flash-Next cells from the max_tokens bug, already rerun at 131072 with --fit-max-tokens. Two are Qwen3-30B-A3B at its native 32k, a deliberate control that stays as it is. The other seven are the table above, and all seven are Claude Code cells. They were rerun on 16 October at 262144, same model, engine, prompt and sampler, one rep each. Strata decodes greedy in both runs, and the Laguna-XS arm keeps llama-server's default sampler.

The reruns at 262144: six PASS, one real FAIL#

Every rerun finished with no context error. The review column is the blind 1–5 rubric from part 13.

• Flash-Next t20 reviewed 4, the best of the seven reruns and level with the study's best so far. Every money probe in the review came back clean, and the run's own claims were honest. At 131072 the same cell was an 85-point FAIL with a missing README.
• The 100-point cells stay 100-point cells. Their reviews are not better for having finished. Swift-1.5's Zig cell still swallows its rollback error with catch {}, and so does the Flash-Next tdd2 cell. Finishing turned a FAIL into a PASS, and the code is what it was.
• Laguna-XS t15 is a real FAIL. Two hidden core tests fail again, and the mutants dropped from 6 of 7 caught to 4 of 7. The transfer saves the source and then the destination with no compensation if the second save fails, while its README promises "Either both accounts are updated, or neither is." That is the one lie among the seven reruns; five overclaim, and t20 is the only fully honest one.

Four of the seven never needed the extra room#

The last column is the surprise. Four reruns peaked below 131072: Swift-1.5 t13 at 110,734, the tdd2 cell at 116,134, Laguna-XS t14 at 113,290 and Flash-Next t20 at 126,310. The bigger window was never touched. Only Swift-1.5 t15, t21 and Laguna-XS t15 went past the old cap, t21 by 35k tokens.

A rerun is a new run. With one rep per cell, the second session took a shorter path in four cells out of seven, and one rep can't say why. So these reruns don't prove that the originals would have passed with more room. They prove something narrower and more useful: none of the seven rerun verdicts was produced by the window. Six say PASS, and the seventh FAIL belongs to Laguna-XS.

Check the cap before grading the code#

The rule on the bench since 15 October is that a FAIL on a Strata arm gets checked for a context refusal before it is read as a code result. Laguna-XS showed the rule was too narrow: llama-server refuses too, with a different string. From here on the rule is the grep above, which covers both, run on every FAIL before it goes into a table.

The fix the other way is just as cheap. Before a model goes into the matrix, read its trained context from the GGUF and serve it there, not at whatever number the serving config inherited.

Stage 2 changes the window and the sampler together#

The stage-2 arms already queued on Strata (90a, 90b, 91a) serve at 262144, and they also set the vendor sampler instead of greedy. That is two changes in one comparison against stage 1. The control costs no GPU time: take each run's peak prompt from its archived session, the same count as the last column above. A run that never passed 131072 is a sampler-only comparison. A run that did is marked and reported separately. The reruns in this post already show why that split matters, since four of seven landed on the sampler-only side without trying.

Filed under: amd, local-llm, ai-agents, claude-code, benchmarks

Read next#

• The best-reviewed code came from the slowest Flash-Next arm — part 13: the stage-1 table and its sampler and context confounds; this post counts how many verdicts the 131072 window decided.
• 128k context is the floor, and prefill is the wall — part 5: where 131072 came from.
• Read the transcript, not the gate — part 9: another verdict that only the transcript could correct.

Sources#

• Agent-matrix rows: a9-results/matrix/results*.jsonl, aggregated with matrix-summary.py for the runtags s1-swifthcq8, s1-vknew, s2-fnjvm, s2-fntdd2 and their 256 reruns, 2026-10-16 (private bench repository). Verdicts, points, hidden tests, mutants, wall times and review scores come from these rows.
• Run archives on the A9: ~/bench-archive/<run-id>/serve.log for the refusal lines and the 12-archive count; the archived Claude Code session JSONL for peak prompt tokens (input plus cached tokens per request, maximum per session).
• Blind reviews and honesty labels: review/reviews/<run-id>.json and the honesty files for the seven reruns (private).
• Laguna-XS-2.1 Q8_0 GGUF header, read on the A9 on 2026-10-16.