The model said it saw no output. It was right.

Assumes you've driven a coding agent against a local llama.cpp server and know what a tool call is.

North-Mini-Code made eight tool calls in a Java task under Claude Code and gave every one of them the same id, 1. Claude Code ran all eight and logged eight real results. The model was shown one of them, a read error. Within a minute it decided the project directory was empty. From where it sat, that was correct.

That bug decided three cells of the agent matrix, one per language, before the model had written a line. Fixing it took one condition in llama.cpp. The code the model then wrote with a working harness scored 1 out of 5 in five blind reviews out of six.

Three Claude Code cells, 29 points, about a minute each#

Stage 1 ran North-Mini-Code Q8 on llama.cpp Vulkan through both harnesses, on the three ledger-transfer tasks: Java (t13), Go (t14) and Zig (t15). The opencode cells did the work: 70 points on Java, 74 on Zig, and 22 on Go, where the old 32,000-token reply cap cut the cell. The Claude Code cells looked like this:

29 is what the scorer gives a repository nobody touched. Three identical scores from three languages, each in about a minute of a slot of one hour (90 minutes for Zig), is a strange row for a model that reached 70 on the same Java task under opencode. It still read as a model result: some models give up.

The Java cell's session log has the giving-up in it. The fourth call is a Bash command, and Claude Code's log records its result: the working directory's path and a total 60 listing. The model's next reasoning block opens: "It seems like I'm not seeing any output."

Side by side, that looks like a model hallucinating an empty directory. The log says the output came back.

That cell had no verbose server log, so it can't show what the model was sent. The next run can.

Every tool call was named 1#

On 19 October I reran the Java cell as a ten-minute diagnostic: same model, same engine, same Claude Code 2.1.290, with llama-server -v so the server log keeps what it sends and what it renders. It also ran the sampler from the model card (temperature 1.0, top_p 0.95, top_k and min_p off) instead of llama.cpp's defaults, which matters later. The run took 68 seconds and made eight tool calls. This is every tool_use block the server streamed back to Claude Code:

$ grep -o '"type":"tool_use","id":"[^"]*"' serve.log | sort | uniq -c
      8 "type":"tool_use","id":"1"

North-Mini writes its own tool_call_id, and it writes 1 every time. llama-server only generates an id when the model's is empty, so all eight calls reached Claude Code under one name.

What Claude Code logged vs what the model was given#

Claude Code's own session log is complete. Eight calls, eight results, and the results are real: the second call, a find for README.md, returned the README's path.

The prompt the model got is a different document. In the last one the server rendered, one call survives, with one result: EISDIR: illegal operation on a directory, from trying to Read a directory. The seven turns after it arrive as reasoning only, each call gone and its result nowhere. Five of the user turns that follow them read (no content); the other two carry a nudge from Claude Code, of which more below. The (no content) placeholder is Claude Code's. The string appears three times in the 2.1.290 binary and in no file under llama.cpp's tools/ or common/ source on the box.

So the history Claude Code kept and the history it sent back disagree from the second call on. The patch I wrote later carries the working theory in its comment: with every id the same, Claude Code treats each later call as one it has already answered. I haven't traced the line in Claude Code that does it. The effect is in the prompt, and it is enough.

The model's reasoning after the second call: "The path provided is pointing to an empty directory." The only output it had been shown was an error. It was right.

Claude Code, for its part, told the model three times in 68 seconds: "[Your previous response had no visible output. Please continue and produce a user-visible response.]"

The fix: one condition in llama.cpp's chat code#

The patch goes into common/chat.h, where llama-server assigns ids to parsed tool calls. Upstream generates one only when id.empty(). The patched condition (comments left out):

auto id = tool_calls[i].id;
if (id.empty() || std::getenv("LLAMA_UNIQUE_TOOL_IDS") != nullptr) {
    id = gen_tool_call_id();
}

With the variable set, the model's id is ignored and every call gets a fresh one. The bench builds it as a separate engine, vulkan-uid, which sets the variable, next to the unpatched builds.

A second ten-minute diagnostic on the patched engine, with the same card sampler as the first, logged 55 tool_use blocks with 55 different ids and not one (no content). The model worked until the wall stopped it at 600 seconds, at 86 points. That is one run, killed by a ten-minute limit: it shows the mechanism is gone, and nothing more. It is not a comparison with the four-hour cells below.

The rerun changed the sampler too#

The real rerun was a card arm: all six cells at four hours, a 65,536-token reply cap and 262,144 tokens served, on the patched engine, with the card sampler. Points per cell, with the best earlier result for each:

All six still FAIL. The Java cell got to 91 in 686 seconds. The Go cell ran into the 240-minute wall.

Between the 29s and these cells the patch changed, and so did the sampler, the wall, the reply cap and the served context. The old cells used none of the wall, cap or context they had: they stopped after about a minute. The sampler is the one that could matter, and the two diagnostics settle it for the stop: the unpatched one already ran the card sampler and still quit at 68 seconds on 29 points. So the Claude Code gains go to the patch, and the sampler gets no credit for them. What stays open is whether the sampler helped or hurt the four hours of work on top. No patched Claude Code cell at llama.cpp's default sampler exists, at any wall.

The opencode column, where the old cells already worked, is where a sampler effect would show, and it shows nothing clean: +9, -21, and a +39 against a one-hour cell that had been cut at the old output cap.

Comparing the columns with each other carries one more confound: every cell on the A9, in both harnesses, loads my personal ~/.claude/CLAUDE.md, and every Claude Code cell also loads my 17 rules files on top of it. An earlier version of this post said opencode never saw any of it. A capture against a fake server showed otherwise: opencode puts the whole CLAUDE.md into its system prompt unless OPENCODE_DISABLE_CLAUDE_CODE=1 is set, and the A9 bench never set it. It skips the rules files. The 3090 bench had a related leak, where Claude Code ran with an empty config directory and OpenCode alone carried the file. All of this is equal across models, so it doesn't move the rankings, but the rules files sit between the two harness columns.

t15 went down: Claude Code's own "Prompt is too long"#

The patch raised t13 and t14. The Zig cell fell from 29 to 22, below the untouched repository's score.

It stopped after 66 minutes on an error from Claude Code: Prompt is too long. The turn before it had a 173,799-token prompt. The server's last line for that turn ends the slot at n_tokens = 176046, out of 262,144 served, and its log has no context error at all. No request after that turn reached the server. Claude Code stopped itself.

Part 14 was about the server refusing requests at 131,072 tokens. This one is the client, with about 86,000 tokens of the served window still free. The line comes from Claude Code itself. For a model id it doesn't recognise, its model configuration docs say it "can't tell the model's context window and assumes 200K tokens". In the 2.1.290 binary, that assumed 200,000 puts auto-compaction at 167,000 tokens and a hard stop at 177,000. The last real request was 173,799 tokens in and 2,248 out. The usage the server reported for that turn, 176,047 tokens, was still under 177,000. Claude Code's estimate for the next request, which adds a byte-based guess for the newest messages, came out over it, and Claude Code wrote Prompt is too long into its own transcript 13 milliseconds after the result of the model's last tool call came back, about a tenth of a second after the model's last reply, without sending anything. The bench never set CLAUDE_CODE_MAX_CONTEXT_TOKENS, the variable that corrects the assumed window. Auto-compaction should have fired by 167,000 on the way up. Why it didn't in this cell, I haven't pinned down. Other cells did compact, at 167,000 to 172,000 tokens. Since 20 October the bench sets the variable to the window the server serves (262,144 for these cells), and four cells that had auto-compacted at the old limit were rerun with it: Ornith t15, Step-3.7 t14 and t15, and Flash-Next's Micronaut cell. No verdict moved: 89 stayed 89, two FAILs stayed FAILs, 100 stayed 100. They are not exact repeats: Ornith's and Flash-Next's reruns had a 4-hour wall instead of 1.5, and Ornith's used 97 minutes of it.

What it left behind was a Zig test suite that doesn't compile: use of undeclared identifier 'test_now_ms'.

A working harness, the same verdict#

Fixing the harness let North-Mini do the task. The blind reviews then scored the code it wrote: 1, 1, 1, 1, 1, and 2 for the opencode Java cell. In all six cells, one idempotency key used on two different account pairs moves money twice, and a storage failure can lose money. In the Java and Zig cells it takes nothing more than a failed second save: the 91-point Java cell leaves from=70.00 to=0.00. The Go cells lose it through a failed rollback or a cancelled context. In four of the six, both Java and both Zig cells, a retry after a failure moves the money again.

Then the cell-specific ones. The Go opencode cell keys idempotency by amount, and a plain replay of a finished transfer crashes the process with sync: unlock of unlocked mutex. The Zig opencode cell deadlocks on opposite-direction transfers. It also committed Remove hanging transfer test, deleting its only transfer test file, and its closing summary then credited tests/transfer_test.zig with "12 test cases covering all transfer scenarios".

The honesty check labels five of the six cells false-claim and one overclaim. One of the five is the Go Claude Code cell that hit the wall, and a false claim in a killed run counts as unproven. That leaves four lie cells, and North-Mini keeps its LIAR flag. The fixed harness let it lie in more detail.

North-Mini's weights came off the A9 on 20 October, so the record ends here. Its best Claude Code score, the 91, arrived eleven minutes into a four-hour window, in code that loses 30.00 when its second save fails.

Filed under: amd, local-llm, ai-agents, claude-code, benchmarks

Read next#

• Read the transcript, not the gate — part 9: a harness verdict that only the transcript could correct.
• Seven FAILs, one setting: rerunning agents at 262144 tokens — part 14: the server-side context cap, where this post's t15 stop is the client-side one.
• How to tell when your coding agent lied, not just failed — part 16: how lie cells and the LIAR flag are counted.
• My benchmark read my CLAUDE.md — the personal-config confound behind every Claude Code vs opencode comparison here.

Sources#

• Run archives on the A9, ~/bench-archive/<run-id>/: the stage-1 Claude Code cells (runtag s1-vkr1, 13-14 October), the diagnostics dg-northclr1 (unpatched, llama-server -v) and dg-northcluidr1 (patched), and the card arm s2-northcr1 (19 October). Manifests for wall times, tool-call counts, points and served context; serve.log for tool ids, rendered prompts and the last slot of the t15 cell; Claude Code session JSONL for reasoning text, results and the Prompt is too long message.
• Claude Code docs, Model configuration: the 200K window assumed for an unrecognised model id, and CLAUDE_CODE_MAX_CONTEXT_TOKENS.
• Claude Code 2.1.290 binary on the A9: the assumed window and the compaction and stop thresholds derived from it; the same binary and the llama.cpp source trees there, searched for the (no content) placeholder.
• engines/llamacpp/unique-tool-ids.patch in the private bench repository (a9-agent-bench).
• Blind reviews and honesty labels for the six card-arm cells: review/reviews/*s2-northcr1.json and review/honesty/*s2-northcr1.json (private).
• Bench plan, North-Mini paragraphs for 89u, 89u2 and 89u3: docs/A9-PLAN.md (private).