Read the transcript, not the gate

The prompt asks for one file, hello.py, containing print('hi'), and then asks the agent to stop. The model is Qwen3-Coder-Next at Q8, served by llama.cpp on the A9, driven by Claude Code. Its reply comes back in one turn. It reports the file made, line and all, and signs off.

The session transcript holds one user message, one assistant message, and nothing between them. No Write. No Bash. No tool call of any kind. No file was written, because nothing ever tried to write one.

The intake directory has a9-mega-qwen3-coder-next-q8.gate-ok in it. That model passed.

What the intake gate checked#

The intake wave ran from 10 October into the morning of the 11th: 17 models, each given the same smoke task under three agent harnesses, opencode, Claude Code and pi. The GATE graded the models on one of them: opencode had to create hello.py inside the project directory. pi and Claude Code were run, logged, and not gated, the rule part 4 quoted from the script's header the day before.

By that rule, 15 of 17 passed. The two that didn't:

• Qwen3-Coder-30B Q8 wrote /tmp/opencode/hello.py, outside the project, on all three tries.
• Ling-3.0-flash never got as far as a prompt. llama.cpp b11538 refused the GGUF:

0.00.256.623 E llama_model_load: error loading model: check_tensor_dims: tensor 'blk.0.ssm_f_a.weight' not found

Fifteen markers, then. The smoke table already printed a FILE column for Claude Code too, and for seven of the eight failing models in the table below it said no. Nothing read it. On the morning of the 11th the Claude Code transcripts got read, all of them, and put a cause next to each failure:

Eight of the fifteen passes had something wrong under Claude Code. The gate saw none of it, because by construction it couldn't. The failures sort into three classes, and each one looks different in a transcript.

Class 1: a chat template that raises on Claude Code's system message#

Four models answered every Claude Code session with HTTP 500. The cause sits in the conversation shape. Claude Code sends a second system message after the first user turn. opencode and pi never send that shape, so neither of them tripped on it. The intake's own template check did. It sends four request shapes to the stock template, a late system turn among them, and for all four models it printed latesys FAIL minutes before their smoke ran. That line sat in the same log as the gate's verdict, unread like the FILE column. The stock chat templates baked into these GGUFs refuse it. The Jinja template raises, llama-server turns the exception into a 500, and the session ends before the model has seen a token.

The Qwen-family templates (Flash-Next, the ISTA Coder build, Qwen3.5-122B) all raise the same sentence: System message must be at the beginning. Mistral-Small-4's Claude Code session hit Only user, assistant and tool roles are supported, got system. A template that raises is a reasonable choice for a model trained on one conversation shape. It is also a hard wall for a client that sends another.

The fix is a per-model template override. A generator, make-template.py, reads tokenizer.chat_template out of the model's own GGUF and patches only the raise. Mistral-Small-4's version renders a late system turn as a [SYSTEM_PROMPT] block instead. Then the same four-shape check, now run against the stock template and the override side by side, ending with the tool-call case:

== flashnext-iq4xs / STOCK template (from the GGUF)
  single     OK    finish=stop  reasoning=122ch
  multiturn  OK    finish=stop  reasoning=57ch
  latesys    FAIL  Exception: System message must be at the beginning.
  tools      OK    finish=tool_calls calls=2 (parallel) bash{"command":"ls /etc/apt"}, read{"filePath":"/etc/hostname"}
== flashnext-iq4xs / OVERRIDE launch/templates/flashnext-iq4xs.jinja
  single     OK    finish=stop  reasoning=122ch
  multiturn  OK    finish=stop  reasoning=57ch
  latesys    OK    finish=stop  reasoning=128ch
  tools      OK    finish=tool_calls calls=2 (parallel) bash{"command":"ls /etc/apt"}, read{"filePath":"/etc/hostname"}
== done (server stopped)

latesys is the Claude Code shape: system, user, system, user. All four models failed it on the stock template and passed on the override, with tool-call output identical to stock. Mistral's stock template failed that synthetic case with a third message, the one about roles that must alternate user and assistant, which is a different sentence from the one its real session hit. One template, two separate checks, both fatal.

The same class came back that evening from a different client. The output-check script sent reasoning_effort: "low", a setting meant for gpt-oss, to Mistral-Small-4. Its template accepts only none or high, raised, and every prompt got a 500. The results file graded the 500s as failed checks, so a broken request looked exactly like a model that couldn't list primes. It now sits in a directory named invalid-20261011-reasoning-effort-500/, and the client retries without the field.

Class 2: success reported, zero tool calls#

Qwen3-Coder-Next and Qwen3-Coder-30B, under Claude Code, made no tool call and answered that the file existed. For the 30B that happened in all three of its sessions: one assistant message each, each a confident one-line report, none backed by a call.

From the outside almost nothing looks wrong. There is no error, no 500, no stray file. The client exits, and the model's last message says the work is done. One check disagreed: the smoke's FILE column said no, because there was no file, and nobody was reading that column. The transcript adds what FILE can't: no tool call happened at all. Whether the model never emitted one or something between the model and the client lost it, the transcript doesn't say. It only shows that no call happened, which is the part that matters for a gate.

Class 3: writing outside the project#

The working directory is not a suggestion every model takes. Four did their writing elsewhere:

• gpt-oss-120b (Claude Code) wrote /home/user/hello.py, the case part 4 already told. Its first opencode intake had also written /tmp/opencode/hello.py.
• gpt-oss-120b-heretic (Claude Code) wrote /home/user/hello.py too, then the project file as well. Its FILE column said yes, so this is the one case only the transcript caught.
• MiniMax-M2.7 (Claude Code) wrote ~/.claude/projects/-tmp-smoke-minimax-m27-iq3xxs-claude/hello.py. That is the directory where Claude Code keeps the session's own transcript. The homework went into the folder the evidence lives in.
• Qwen3-Coder-30B (opencode) chose /tmp/opencode/hello.py, three tries out of three.

Four models, under opencode and under Claude Code. That spread points at model behaviour rather than a bug in either client, and it is the class most likely to hurt outside a smoke test: a coding agent that edits the wrong tree passes its own task and breaks someone else's.

Damage on the harness side#

The models weren't the only ones leaving things behind. The chat-template check starts its own llama-server for each intake. During one intake that server was never stopped, and the queue runner refuses GPU jobs while a foreign process holds the GPU. The next intake in line logged this at 06:06:

{"ts": "2026-10-11 06:06:26", "job": "65-gpu-intake-z-step37-flash-iq3xxs.job", "event": "gpu-wait", "pid": 46458, "ppid": "1", "cmd": "/home/user/tools/llamacpp-vulkan-master/llama-server --chat-template-file /home/user/work/a9-agent-bench/launch/templates/qwen36-35b.jinja --host 127.0.0.1 --jinja --log-colors off --port 45541 --sleep-"}

ppid 1: an orphan, and its template file names the previous intake's model. The runner waited 3.7 hours, 06:06 to 09:50, for a process nothing in the queue was going to stop. Killed by hand, it ignored SIGTERM and needed SIGKILL. The safety rule did its job. The leak was the bug. Every intake job now kills leftover servers, TERM first, KILL five seconds later.

The archiver had a quieter failure. It exported opencode sessions with opencode session list and opencode export. The opencode CLI docs describe session list as "List all OpenCode sessions." It lists the sessions of the current directory's project: a git repo is its own project, a plain directory counts as "global". Run from any other directory, the archiver found no sessions and exported nothing, without an error.

That matters because the audit only worked while the transcripts still existed. Graded runs used to fare worse: the bench ran rm -rf on a cell's workspace at every re-run, so a re-run left one line in a results file and nothing to read. Every agent run is now archived whole, per model, harness, task and options: the workspace and its git history, the session logs of whichever harness ran (opencode, Claude Code, pi or omp), a slice of the server log, the prompts, and a manifest with token sums.

What a gate has to see#

A smoke gate that reads one harness's output file answers one question: did this harness, with this model, produce this file? For the four 500 models the answer under opencode was yes, and under Claude Code the model never saw the prompt. Both are true, and both were in the log. Only one of them was being read.

The audit points to three conditions, per harness, all of them cheap once the transcript is on disk:

• the session completed without a server error;
• at least one tool call happened;
• every file the agent wrote is inside the project directory.

None of the three is in the gate yet. As of this writing the intake script still checks one thing, the opencode file; the other columns get read by hand before a model is ranked.

print('hi') asked nothing of these models except to put eleven characters in the right place with a real tool. Eight of the fifteen that passed missed on at least one harness. The next part puts the survivors in front of tasks long enough for the code itself to be wrong.

Filed under: amd, local-llm, ai-agents, claude-code, testing

Read next#

• The download that contaminated the benchmark — part 4: the queue that runs these intakes, and the first time Claude Code wrote hello.py into the home directory.
• 128k context is the floor, and prefill is the wall — part 5: what these same models cost once the context is long enough for real agent work.

Sources#

• opencode CLI documentation — session list ("List all OpenCode sessions.") and export, as fetched 13 October 2026.