The harness decided how long the model thought

Assumes you've run a coding agent against a local llama.cpp server and know what prefill, decode and reasoning tokens are.

On the first run of t11, a Bash script that files an inbox of messy files by what they contain, Qwen3.8-27B wrote 62,977 characters of reasoning. The run took 425 seconds and passed. On the first run of the same task in Pi, against the same server settings, it wrote 8,911 characters, took 90 seconds, and passed.

That was not one strange pair. Over all six tasks and three runs each, the base Qwen3.8-27B averaged 250 seconds per run in OpenCode and 116 in Pi. Both arms passed 18 of 18. The serve block each result row records, engine build, weights hash, sampler down to top_p, is identical between them. Whatever doubled the time came in through the client. That includes a top_p override I found later, and a follow-up showed the override isn't what doubled it.

The clean rerun: 252 runs, and nothing to rank but time#

This was the first full pass after the 15 October harness fixes: MTP drafting on in the bench server, OpenCode isolated from my ~/.claude, run logs out of the workspace. Seven models, six tasks (three Go/Java dev tools with tests, then a Python CSV tool, a Bash file sorter and a Node JSONL merge), three runs each, both harnesses. 252 runs between 12:00 and 21:59 on 16 October, no flags for peeking at the answer key or escaping the workspace.

s/run is the mean wall time per run. pp and tg are server tokens per second from llama.cpp's per-request timing lines: prefill over requests with at least 1k new prompt tokens, generation over replies of at least 32 tokens. Ornith's server drafted three tokens where my launcher drafts one, so its tg and acceptance are not the launcher condition; its verdicts stand.

All seven misses are the model's own logic: skipped-row counting on the CSV task, branches listed in the wrong order, a CSV without a trailing newline filed as something else, one rounding detail. Four models passed 36 of 36. With three runs per cell, nothing separates them on correctness, so time is the only ranking left. Under Pi, the four Qwen3.8 27B tunes span 29.5 seconds per run, swift15's 94.9 to thinkingcap's 124.4. Moving qwen38 base from Pi to OpenCode costs it 134.

What the two qwen38 arms share#

The server side is identical. The differences sit in the client:

• Server: one llama.cpp start per arm, --reasoning-effort medium, 128k context, the launcher's chat template, MTP with three draft tokens. The rows' serve blocks compare equal.
• Clients: OpenCode 1.18.4, isolated, and Pi 0.87.1 with its isolation flags. Pi sends no sampler fields. OpenCode 1.18.4 sends top_p: 1 whenever the model id contains "qwen", a default built into that version, so qwen38 and qwen38pi sampled at 1 under OpenCode and at the server's 0.95 under Pi. swift15 and thinkingcap got no sampler field. My 14 October request capture missed this: its OpenCode run used an id without "qwen". OpenCode asks for up to 32,000 output tokens, Pi 32,768. Pi's catalog marks the model reasoning: false and OpenCode's sets no reasoning option, which leaves reasoning to the server flag.
• Pi's --thinking off: inert against llama-server. Pi logs a thinking-level change and then stores whatever reasoning the server streams. The sessions from this run are full of it.

The prompt-size explanation: true, and worth 11 seconds#

The suspect I reached for first was the system prompt. OpenCode's, isolated, is 10,785 characters with nine tools. Pi's is 2,744 with four. That capture is where I found my CLAUDE.md riding along in OpenCode.

The server logs show where it lands. On qwen38's first run, OpenCode's opening agent request was 7,823 tokens and Pi's was 1,981. After the opening request, llama.cpp's prefix cache serves everything already seen, and the turns cost the same in both clients: a median of 125 new prompt tokens per OpenCode turn, 126.5 per Pi turn. On qwen38 the prompt difference is paid about once per run, cold.

Summing every prompt eval time line in each arm's server log and dividing by 18 runs: OpenCode spent 16.5 seconds per run on prefill, Pi 5.5. Eleven seconds of a 134-second gap.

Server time, prefill plus generation, is 94 to 97 percent of the wall in every 27B arm but one. The missing 123 seconds were not spent in the shell or in the client. They were spent generating.

Where the other 123 seconds went: generation#

Per run, qwen38 generated 15,020 tokens under OpenCode and 7,642 under Pi. It also decoded a little slower in OpenCode, 66.3 tokens per second against 73.7, with lower MTP acceptance. That slower rate is worth about 23 seconds. The other 100 come from the count: give OpenCode Pi's decode rate and generation still takes 100 seconds longer.

The sessions say what those tokens were. OpenCode stores each turn's parts in a per-run SQLite database, Pi in a JSONL session file, and both keep reasoning apart from visible text and tool calls. Counting characters, which tracks the server's token totals closely (2.04x against 1.97x for qwen38):

Tool-call arguments are the commands the model ran and the code it wrote into files. Those are close. The reasoning is 3.6 times larger. Under OpenCode the base model spent 3.6 times as many characters thinking about roughly the same amount of code.

It is not one runaway session. Across the four 27B tunes, OpenCode's mean reasoning per run beats Pi's in 23 of 24 task cells. Per model, the ratio of mean reasoning runs from 1.8x for qwen38pi to 3.8x for thinkingcap. On t11 qwen38 reasoned 62,977, 40,971 and 45,393 characters in its three OpenCode runs, and 8,911, 5,378 and 9,053 in Pi. The qwen38 and qwen38pi OpenCode runs also had top_p 1; swift15 and thinkingcap, sampled alike in both clients, lean the same way.

Muse and Laguna think more in Pi#

If that were all, the lesson would be "Pi makes models terse". The other two reasoning models disagree. Muse-glimmer reasoned 24,140 characters per run in Pi against 12,737 in OpenCode. Laguna: 17,878 against 9,250, with 44 turns per run against 31. Both reason more under Pi in all twelve of their task cells.

They are also the only two models that were slower in Pi. Muse took 252 seconds per run there against 219; Laguna 144 against 92. Same mechanism, opposite sign: for each model, the harness that drew out more reasoning was the slow one.

The split carries a confound this data can't remove. Only the four 27B tunes ran with --reasoning-effort medium; Muse and Laguna ran their model cards' defaults. The numbers support "OpenCode draws more reasoning out of Qwen3.8 tunes at medium effort". They don't support "OpenCode draws more reasoning". Ornith ran with reasoning off and logged none in either harness.

What I tested: replay, top_p, prompts, tools#

The follow-up started with a logging proxy between each client and a server on the same settings, one t12 run per client. It settled one suspect and turned up another:

• Reasoning replay is the same. Both clients send every earlier assistant turn back with its reasoning_content, and the template keeps it.
• The sampler is not. Every OpenCode request carried the top_p: 1 above.

The rest took 90 runs on 18 October: qwen38 and swift15, the three scripting tasks t10 to t12, three runs each. The base and swapped-prompt arms alternated OpenCode and Pi per model, where on 16 October every OpenCode arm had gone first; the top_p and four-tool arms ran last. All 90 passed. Mean reasoning characters per run, n = 9 per arm:

The gap follows the client. On mean, every OpenCode arm out-reasoned every Pi arm, for both models, whichever prompt it carried. Matching top_p took qwen38 from 14.8k to 13.3k; cutting to Pi's four tools left 22.1k. Each swapped prompt describes the other client's tools, but neither swap shows tool misuse: no calls to missing tools, only shell commands exiting non-zero, as in the base arms.

Noise is the other result. Single runs span about 1k to 70k characters, and the unchanged OpenCode setup gave qwen38 27.3k on these three tasks on 16 October and 14.8k here. Nine runs per arm separate OpenCode from Pi, not the OpenCode arms from each other: 27.0k on Pi's prompt is no evidence that Pi's prompt adds reasoning. And the extra reasoning bought nothing: qwen38's OpenCode arms averaged 119 to 218 seconds per run, its Pi arms 66 and 91, and every run passed.

So the mechanism is narrowed, not found. What's left untested is what every OpenCode arm shared:

• Tool schemas. 19,814 characters of JSON against Pi's 2,956. The four-tool arm dropped tools but kept OpenCode's schema wording.
• Message formatting. How OpenCode wraps tool results and earlier turns on the way back.
• The env block OpenCode appends even to a replaced prompt. The skill list it adds is out: the four-tool arm lost it and still reasoned 22.1k.

What the speed ranking is worth now#

swift15 is the fastest 27B tune by mean in both harnesses: 118.5 seconds per run in OpenCode, 94.9 in Pi, with qwen38pi 5.7 and 0.5 seconds behind. Below that, the order changes. Under OpenCode, qwen38 base takes 2.1 times swift15's time and finishes last, behind thinkingcap. Under Pi it takes 1.2 times swift15's time and finishes ahead of thinkingcap.

That thinkingcap number carries its own asterisk. One of its Pi runs on t7 took 731 seconds, and 600 of them were a single tool call: the model ran its test suite, a test it had written hung, and Go's default ten-minute timeout ended it with panic: test timed out after 10m0s. The row says PASS. Without that run, its Pi mean is 88.7 seconds; the other arms keep their slow runs, so that figure ranks nothing.

Two more limits. Every OpenCode arm ran before every Pi arm, in one queue on one afternoon and evening; the follow-up, which alternated them, kept the direction. And three runs per cell is enough to see qwen38's 2.2x gap in mean wall time, not enough to order swift15 and qwen38pi, 0.5 seconds apart under Pi.

The prefill post closed on output tokens per task as the number nobody adds up. Added up here, they decided the order of the 27B tunes, and for qwen38 the harness alone accounted for almost half of OpenCode's 15,020.

Filed under: local-llm, llm, ai-agents, benchmarks, opencode

Read next#

• 2.2x faster prefill, 1.04x faster agent — the same server-log accounting on an engine comparison: prefill was a small share there too, and the faster engine's arm wrote 1.7x more tokens.
• My benchmark read my CLAUDE.md — the request capture behind the system-prompt sizes above, and why OpenCode needed isolating.
• My benchmark served the model at half speed — the MTP fix that made this rerun necessary.

Sources#

All from my local benchmark repository (not public), under agent-playground/bench/:

• forensics/rebaseline-20261016/README.md and pp-tg.txt: conditions, the representative table, per-arm server speeds, and the Ornith draft-count deviation.
• results-rb11.jsonl to results-rb13.jsonl (OpenCode) and results-pi-rbpi11.jsonl to results-pi-rbpi13.jsonl (Pi): 252 result rows with wall time, verdict, client and serve blocks.
• serve-logs/<model>-rb1.log and -rbpi1.log: llama.cpp print_timing lines, summed for prefill and generation seconds and tokens.
• Per-run sessions: OpenCode's opencode.db in each run's metadata directory and Pi's session JSONL, for reasoning, text and tool-call character counts.
• forensics/payload-20261014/README.md: the client request capture (sampler fields, max_tokens, system prompt length, tools); its no-sampler finding held only for the model id it captured.
• .queue34-rebase-20261016.sh: the queue, with per-model server settings and arm order.
• forensics/harness-reasoning-20261018/README.md, proxy.py and the capture-*.jsonl files: the logging-proxy capture (top_p: 1, reasoning replay, prompt and schema sizes) and the follow-up's arms.
• forensics/harness-reasoning-20261018/analyze.py and analysis-20261018.txt, over results-hr* and results-pi-hr* and their sessions: the follow-up's 90 runs, same character measure as above.
• .queue36-harnessreason-20261018.sh: the follow-up queue.