The template differed. The prompts didn't.

· 10 min read
Contents

Builds on Read the transcript, not the gate, where a chat template broke every Claude Code session on four models.

The plan had one variable. Same model, two quantizations, only the weights differ, and I was about to run it on Ornith-1.5-122B, a 122-billion-parameter mixture-of-experts coding model, on my 128 GB AMD Strix Halo box. On one side, the 4-bit UD-Q4_K_XL from unsloth, a team that publishes quantized builds of many big open models, about 80 GB. On the other, a GSQ-RCO IQ3_S from the Hugging Face account fajardc, about 53 GB, a mixed quant whose tensors range from IQ2_XS to Q6_K. Ten coding-agent runs each: five tasks in five languages, each through two agent harnesses, opencode and Claude Code, the programs that turn a model's replies into file edits and shell commands.

I stopped trusting the word "only" on this box a while ago; around here it usually means "and one more thing nobody wrote down". A GGUF, the model file format llama.cpp loads, carries its own chat template in its metadata, the Jinja program that turns a list of messages into the text the model actually reads, and llama-server, llama.cpp's HTTP server, renders whatever the file brings. My bench reads that template before a model gets anywhere near a run. In part 9 that check printed the failure minutes before the smoke test and I didn't read the line; I read all of them now. These two templates did not agree. On paper, the quant A/B was also a template A/B.

One hunk at line 100#

The whole diff between the two stock templates is one hunk:

--- ornith122-q4kxl.stock.jinja	2026-10-29 14:20:41.103497415 +0200
+++ ornith122-gsq.stock.jinja	2026-10-29 14:20:40.842765846 +0200
@@ -97,7 +97,11 @@
             {%- endif %}
         {%- endif %}
         {%- set reasoning_content = reasoning_content|trim %}
-        {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
+        {%- if loop.index0 > ns.last_query_index %}
+            {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
         {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
             {%- for tool_call in message.tool_calls %}
                 {%- if tool_call.function is defined %}

The Q4_K_XL template renders the model's earlier reasoning, its <think> block, on every assistant turn it is sent back. The IQ3_S template renders it only on turns after the last user query and drops it on the turns before.

Both harnesses do send earlier reasoning back. Part 12 of the 3090 bench found that for opencode, and on this box a live capture showed it again for opencode 1.18.35, the version these runs used, while a verbose server log showed Claude Code's thinking arriving at llama-server as reasoning_content. So that's how I read the diff on the 29th: the one thing the second template drops is the one thing both clients make a point of sending. I had the headline before the first run had started, and I liked it a lot more than the evidence had earned.

I wanted a culprit, so I hashed the files against their parents. The Q4_K_XL template is the Ornith lab's own chat_template.jinja, byte for byte. The IQ3_S template is byte for byte the template of Qwen3.5-122B-A10B, the base model Ornith is built on. Neither quantizer touched a line. The lab took one condition out of Qwen's template, and the IQ3_S file carries Qwen's version with the condition still in. That left me with a confound and no villain, the least satisfying combination on offer. The lab's model card never says what it changed. One footnote, attached to its Claude Code Terminal-Bench numbers, says "Again, Qwen chat template needs to be modified." It doesn't say which change it means, and the lab's own template still refuses Claude Code's late system turn (a system message after the task, more on that below), so it may not mean line 100. The IQ3_S card doesn't contain the word "template" at all.

Held equal, by construction#

Both arms served one file: the lab template plus one bench patch. Both stock templates raise System message must be at the beginning. when a system turn arrives after the first user turn, and Claude Code sends one, so the bench swaps that raise for a plain system block. Nothing else in the served file differs from the lab's template. All 20 cells (a cell is one task in one harness), ten per quant, report the same served template from the server: 56fef4d65f1dde91 by hash, 7,840 characters. Between the two runs the bench changed only its job list and an intake marker. No harness, scorer, prompt or template changed.

That should have been the end of it. Then I did the thing I should have done before I let a diff write a title: I asked what "the last user query" is in an agent run.

Which query, exactly?#

ns.last_query_index comes from a loop near the top of the same template:

{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
{%- for message in messages[::-1] %}
    {%- set index = (messages|length - 1) - loop.index0 %}
    {%- if ns.multi_step_tool and message.role == "user" %}
        {%- set content = render_content(message.content, false)|trim %}
        {%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
            {%- set ns.multi_step_tool = false %}
            {%- set ns.last_query_index = index %}
        {%- endif %}
    {%- endif %}
{%- endfor %}

It walks back to the newest message with role user that isn't a wrapped tool result. A tool message never counts, and neither does a system one. So: in a coding agent, how many real user messages are there?

One. All 20 sessions, both harnesses, both quants, hold exactly one user message with text: the task. opencode sends tool results back as tool messages. Claude Code sends them as tool_result blocks, which llama-server turns into tool messages, and its per-round token reminder arrives as a system turn. A verbose server log from an earlier run, with the same Claude Code version but a different model and template, shows the shape: one prompt with 53 assistant turns and exactly one user turn, each tool result followed by a system turn, almost always the one that says how many tokens are left. The task is the first user message, and every assistant turn comes after it. loop.index0 > ns.last_query_index holds on every one of them, and the IQ3_S template keeps every reasoning block, same as the lab's.

I have read enough Jinja to know I read it the way I want to, so I rendered both templates over those three message shapes with Python's jinja2. That is not llama.cpp's own template engine, so this checks the templates, not the server:

== templates as shipped
  opencode (1 user turn, tool results): lab 3 of 3 reasoning blocks, gsq 3 of 3 reasoning blocks -> identical
  claude-code (1 user turn, late system per round): lab raises, gsq raises -> identical  [ERROR System message must be at the beginning.]
  chat (a second user turn after 3 rounds): lab 4 of 4 reasoning blocks, gsq 1 of 4 reasoning blocks -> DIFFERENT
== templates with the late-system patch
  opencode (1 user turn, tool results): lab 3 of 3 reasoning blocks, gsq 3 of 3 reasoning blocks -> identical
  claude-code (1 user turn, late system per round): lab 3 of 3 reasoning blocks, gsq 3 of 3 reasoning blocks -> identical
  chat (a second user turn after 3 rounds): lab 4 of 4 reasoning blocks, gsq 1 of 4 reasoning blocks -> DIFFERENT

The chat row is the control, and the rule bites there: give it a second user message and the IQ3_S template keeps one reasoning block of four. The agent rows are identical. As shipped, opencode's requests would have rendered to the same prompt under either template, and Claude Code couldn't have run on either.

So I found a confound, held it equal on both arms, wrote it into a title, and the traffic had never heard of it. I'm keeping the fix. It cost nothing, and I'd rather be thorough about nothing than casual about something.

The A/B, with the template held equal#

Same engine (llama.cpp ROCm b11541), same 262,144-token context, same sampler, one repetition, a 90-minute wall per run. Ten cells per quant: Java, Go, Zig, TypeScript and Python tasks, each through both harnesses. Points are a 0-100 mechanical score. Failure-path tests are hidden tests the agent never sees, with a fake store that fails saves on purpose. A blind review scores each cell from 1 to 5 as a merge decision. A lie cell is one that claims work with no code behind it, by part 16's rules; code that exists and is wrong counts as a bug.

IQ3_S (about 53 GB)UD-Q4_K_XL (about 80 GB)
PASS / PARTIAL / FAIL5 / 4 / 15 / 1 / 4
Points, mean (range)98.3 (94-100)92.5 (40-100)
Killed at the 90-minute wall02, both Zig
Failure-path tests passed41 of 5438 of 54
Blind review, mean2.502.20
Cells with a lie11

The smaller file did not lose. Ten cells a side at one repetition can carry that sentence and not one word more, and most of the points gap sits in one cell. The Zig run under Claude Code on Q4_K_XL hit the wall at 40 points and passed 1 of 7 failure-path tests. Take it out and the other nine Q4_K_XL cells average 98.3, the IQ3_S mean to the decimal. Those nine still include the opencode Zig run, which hit the wall too, at 99 points. Take the Claude Code Zig cell out of both sides and the failure-path count is 37 of 47 on each.

The FAIL column leans on the clock as well. Two of the four Q4_K_XL FAILs are the two wall kills, and my rule nine from part 27 still holds: a run killed at the wall records when the time ran out and grades nothing. The 0.30 review gap is one repetition wide, and I'm not going to lean on it any harder than that.

The rule that never fired#

What I never ran is the IQ3_S with its own template. For runs shaped like these, the render says that arm would have been served the same prompts, so it would come back a resample. The rule matters where a second user turn exists: a chat, or a harness that sends follow-up prompts. That's a different benchmark from this one, and I haven't run it.

So my numbered rules get a tenth, and it has two halves. Read the template before the download, yes. Then read the traffic before the template gets to write the headline. The diff was right about every byte in both files. It just had no idea what the clients send, and for a while neither did I, which the diff at least had an excuse for.

Filed under: amd, local-llm, ai-agents, benchmarks

Sources#