Two rounds, seventeen arms: the engine set the speed, not the quant

Builds on Five languages, two reps: the round-two ranking, which ranks these arms by the quality of their code. This post is about how fast they got there.

Qwen3.8-Flash-Next, a mixture-of-experts model, ran at two quantizations (a quant sets how many bits each weight gets) in round two of my study of local models as coding agents, six model setups on the same coding tickets on my 128 GB AMD Strix Halo box. At Q2_0, 2.40 bits per weight, it generated 58.9 tokens per second; at IQ4_XS, 4.25 bits per weight, 26.6. A 66.4 GB file beating a 93.7 GB one by 2.2x is what a smaller quant is supposed to do: fewer bytes to read per token, more tokens per second. There should be a rule that a speed gap lists everything that changed before it is allowed an explanation. This one listed one thing.

The two arms (an arm is one model with its quant and its engine) also ran on different inference engines, the servers that load the weights and generate the tokens: Strata, an engine for this one model family, served Q2_0, and llama.cpp served IQ4_XS. In round one, an earlier round of 11 arms on the same box, the same IQ4_XS file, served by Strata instead of llama.cpp, ran 1.73x to 2.05x faster. With the engine build held still, swapping IQ4_XS for Q2_0 bought 3 to 11 % in those agent runs. Most of the 2.2x belongs to the engine, and the quant gets a rounding error's worth of credit.

The box, the study, and where these numbers come from#

The A9 is my 128 GB AMD Strix Halo test box (a GEEKOM A9 Mega: Ryzen AI Max+ 395, 128 GB of soldered LPDDR5X-8000). The study running on it asks whether a model served locally can do a developer's job as a coding agent. Each model, with its quantization (quant: how many bits each weight gets) and its inference engine, is an arm. An arm works through coding tickets in two agent harnesses, opencode and Claude Code, the programs that turn a model's replies into file edits and shell commands. One ticket in one harness is a cell.

• Round one (part 13): 11 arms, 6 cells each (a Java, a Go and a Zig ticket, once per harness), 2026-10-12 to 2026-10-14.
• Round two (part 27): 6 arms, 24 cells each (five languages, two reps per harness, plus two wider tickets), 2026-10-25 to 2026-10-30.

That makes seventeen arms. None of this was a speed benchmark. Every speed number in the run tables comes from the inference server's own log of those agent runs: 5,260 requests in round one and 8,933 in round two, one row per request, each row carrying the run and the log line it was parsed from. I no longer believe a tokens-per-second figure that can't name its log line, so every one of these can.

Two engines served them:

• llama.cpp, as llama-server: b11538 with the Vulkan backend for most arms, b11541 with ROCm (in a container) for three round-one arms, a build of PR 30105 for Mistral-Small-4, and b11670 Vulkan for K2-Horizon in round two.
• Strata, an engine for one model family, Qwen3.8-Flash-Next ("Flash-Next" below), a mixture-of-experts model. Every Strata arm ran vanilla Strata v0.1.42 (commit 61b3fb5), built from that tag with its HIP backend. Part 8 is where it came in. Its hcq8 and fast modes are environment switches; fast is hcq8 plus a set of extra ones.

The speed column throughout is the median generation rate over replies of at least 200 tokens, which keeps short tool-call replies out. The prefill column is the median rate of full prefills: requests that read a prompt of 8,192 tokens or more from an empty cache.

The 2.2x, and the four things that changed with it#

Round two's Flash-Next pair, from the server logs of 24 cells each:

• Flash-Next UD-IQ4_XS on llama.cpp Vulkan b11538 with an MTP draft head (multi-token prediction: a small head proposes two tokens, the model checks them): 26.6 tok/s, 18.27 hours of summed cell wall.
• Flash-Next GSQ-RCO Q2_0 on Strata 0.1.42 hcq8 with Strata's own speculation (--spec 4 plus an MTP draft): 58.9 tok/s, 8.21 hours.

The walls track the generation rate. Q2_0 generated 1,358,263 tokens, IQ4_XS 1,448,476, 7 % more, so output volume moves the ratio very little. Both ratios come out at 2.2x.

What the two arms share: 262144 tokens of context, the sampler from the model card (temperature 1.0, top-p 0.95, top-k 20), the same tickets. What differs, four things at once:

1. The weights file. UD-IQ4_XS by unsloth (4.25 bits per weight) against GSQ-RCO Q2_0 by ISTA-DASLab (2.40 bits per weight). That is a different quant and a different quantizer.
2. The engine. llama.cpp against Strata.
3. The decoding scheme. llama.cpp's MTP draft, at most 2 tokens ahead, against Strata's --spec 4 --spec-min-p 0.5 with its own MTP draft.
4. The KV cache type. f16 on llama.cpp, int8 on Strata.

Nothing inside round two separates those four. Round one can, partly.

Same file, two engines: where the 2.2x went#

Round one ran the same IQ4_XS GGUF file on both engines (Strata serves it through its own pack built from that file), over the same six cells. That is the closest thing the study has to an engine A/B. A second pair holds the engine build still and changes the file, along with one engine cache setting. All of it is agent-run data, generation medians at matching context depth:

Multiply the round-one factors and you land near the round-two gap: 1.73 x 1.03 = 1.78 at 32-64k (observed 1.88), 2.05 x 1.11 = 2.28 at 64-128k (observed 2.20). The engine side carries most of it. The weights file, together with an expert-cache setting that differed between those two arms, carries 3 to 11 % in these agent runs.

One word needs pinning down. Engine, decoding scheme and KV type moved together in every one of these comparisons, and no log line in either round splits them. So when this post says "engine", it means that bundle: engine + its decoding scheme + its KV type. I'd love a narrower word. The data isn't offering one.

Two more caveats sit on the round-one factors. The engine row also changes context (262144 against 131072) and sampler: round one sent no sampler settings, so Strata decoded greedy while llama-server sampled with the GGUF file's own keys (temperature 1.0, top-k 20, top-p 0.95, plus its default min-p 0.05). And each round-one arm is six cells, one rep each. Round two fixed the sampler and quadrupled the cells, but by then the two engines no longer shared a file.

The weights factor is never the quant alone either. Q2_0 and IQ4_XS come from different quantizers, and the two round-one Strata arms did not share every engine setting: IQ4_XS ran with --expert-cache 17712 --resident-budget-gib 55, Q2_0 with --expert-cache auto (from Strata's own engine log). An expert-cache size can move decode speed on a mixture-of-experts model. The honest sentence is "this Q2_0 file, from this quantizer, on this engine with its own cache setting, was 3-11 % faster than that IQ4_XS file".

Seventeen arms, both rounds#

Every arm, both rounds. Model names link to the Hugging Face repo the weights came from. Engine carries build, decoding and KV type. Prefill numbers for Strata are approximate: Strata logs prompt time in whole seconds, a resolution I'd accept from a kettle, so each one carries about ±1 s (about 4 % on a 17.7k-token prefill). Review is the blind review mean on a 1-5 scale, from part 13 (round one) and part 27 (round two, core cells); compare it within a round only.

Reading it:

• Speculative decoding is in some rows and not others. The Strata rows and the two Flash-Next llama.cpp rows (MTP n2) speculate, and that inflates their tok/s. Ornith, K2-Horizon and every other llama.cpp arm decode one token at a time.
• Round one is 6 cells per arm, round two 24. A round-one median rests on 97 to 341 replies.
• Round one leaves out five superseded cells (reruns replaced them, as in part 13), and the medians here use the reruns; part 13's wall column also counts the superseded time.
• Round one sent no sampler settings: Strata decoded greedy, llama-server sampled with each GGUF's keys or its own defaults. Round two set the card sampler on every arm (K2 at top-k 0).
• Every Strata arm ran with the hipBLASLt tuning table for this GPU, the setting Strata issue #4034 says is worth about 20-25 % on prompts (measured there on a later engine version, from single runs). Each Strata config set STRATA_HIPBLASLT_TUNING, and Strata's own engine log (not the bench serve log) prints that the table loaded at every start.

Across the 17, generation runs from 63.7 to 10.3 tok/s. In round two alone the spread is 5.7x, Q2_0 over K2-Horizon.

Depth: llama.cpp slows down as the context fills, Strata much less#

A coding agent's context only grows. Most round-two requests landed between 32k and 128k tokens deep, and that is where the engines part ways.

From the 32-64k bucket to 128k+, round two:

ISTA-Coder stops at 64-128k, because its 131072-token context capped its deepest request at 115,192 tokens; from 32-64k to there it lost 1 % (46.0 to 45.4).

Round one shows the same split on one file: IQ4_XS on Strata hcq8 went from 59.2 to 54.6 between the 32-64k and 64-128k buckets (-8 %), on llama.cpp with MTP from 34.2 to 26.6 (-22 %).

The Strata arms are not flat everywhere. Their 8-32k bucket sits high (Q2_0 78.3, Swift 62.0) and wide: Q2_0's p10-p90 there runs 48.5 to 82.2, over 27 replies. From 32k on, the lines nearly level out.

On the IQ4_XS llama.cpp arm, part of the slide is visible in the draft head's own numbers. The MTP draft acceptance falls with depth, from 0.967 at 8-32k to 0.743 at 128k+, so fewer drafted tokens survive, on top of whatever the depth itself costs.

K2-Horizon falls hardest: 29.9 tok/s in its 0-8k bucket (10 replies), 6.6 at 128k+ (14 replies). Its entry in my model list calls it "full attention", which fits the curve, but that is a description, not something these logs measure.

Where the wall time went: generation, and almost nothing else#

Part 5 called prefill the wall: filling a cold 128k context took gpt-oss-120b 29 minutes. In an agent run the wall has a door, and it is the prefix cache.

On the round-two llama.cpp arms, 97.6 to 98.3 % of all prompt tokens came from the reused prefix (IQ4_XS 98.3 %, Ornith 98.2 %, K2 97.6 %). Most requests read only what the previous turn appended. The median request waited between 1.0 s (Ornith) and 4.1 s (K2) for its prompt. Full prefills are rare: 28 to 43 per llama.cpp arm across 24 cells, mostly the first request of a cell and Claude Code's context resets. Strata logs no cached-token count, so its share can't be computed the same way.

So the hours went to generation. Summed generation time against summed cell wall, round two:

Strata prompt hours are approximate, from the whole-second clock. The remainder is time outside logged requests: 0.13 to 0.32 hours on most arms, more on the two arms with cells killed at the time limit, because the request in flight at a kill never logs its timings (K2, 13 kills: 1.39 hours; Ornith, 4 kills: 0.57 hours).

Generation took 74.9 to 90.2 % of the wall in every round-two arm, and 70.0 to 90.1 % in round one. For finishing sooner, the number that matters is the generation rate at 32k and deeper. The full-prefill column, the one I spent part 5 worrying about, mostly decides how long the first request of a cell takes.

Smaller findings#

• Strata fast against hcq8 is the cleanest A/B in the data. Round one, same file, engine, context, decoding, KV and greedy sampling; only the mode switches differ. Fast reads a cold prompt 1.57x faster (~1085.2 against ~692.7 tok/s) and generates at the same rate (57.9 against 58.3). One caveat beside it: five of the fast arm's six cells ran on a server started 1 minute 49 seconds before the commit that added --fit-max-tokens (the manifest records the later commit), so those five ran without the flag; the hcq8 arm had it on two of its six cells. The flag is the fix for Strata's HTTP 400 when prompt plus max_tokens exceeded the context.
• The slowest Flash-Next arm reviewed best, in both rounds. IQ4_XS on llama.cpp with MTP: 28.1 tok/s and a 3.17 review mean in round one, 26.6 tok/s and 2.85 in round two, the top review mean of its round both times. Round one had the sampler confound; round two removed it and the order held. Engine, weights file and run-to-run noise are still tangled in it, and part 27 covers how much one rerun moves.
• Ornith ran at the same speed in both rounds. 34.8 tok/s over 6 cells, then 34.1 over 24; full prefill 1125.8, then 1078.8. Same file, same build b11538, same 262144 context; the sampler differed only in min-p (0.05, then 0). Four more Ornith reruns with a 4-hour wall after round two, same build, file and flags on new server starts, came in at 30.7; all four were Claude Code cells and ran deeper (median prompt 129,241 tokens).
• The Patience Award goes to K2-Horizon: 10.3 tok/s, the slowest of all 17 arms, 35.25 hours of summed cell wall, 13 of 24 cells killed by the clock. Against Ornith on the same backend it runs 3.3x slower. Model, architecture, file, build (b11670 against b11538) and sampler all differ, and K2 has no synthetic benchmark to cross-check, so 3.3x is a fact about this setup and nothing narrower.
• gpt-oss-120b spent the least time by producing the least. 33.9 tok/s, mid-table, but 76,566 generated tokens in six cells (Ornith: 399,177) and 0.80 hours of wall.
• Mistral-Small-4 had the slowest full prefill of all 17 arms, 179.8 tok/s, on the PR 30105 build from part 7.
• Harness: same rates, different depth. Within every round-two arm, opencode and Claude Code generation medians sit within 2.4 tok/s of each other. Claude Code ran deeper (median prompt 104,810 tokens against 85,120 on Ornith).

A synthetic cross-check, kept apart#

Separate method, separate table. Between 2026-10-11 and 2026-10-14 my measure.py script timed one long request per prompt size (3 runs, warm medians, thinking off, at most 1,024 output tokens) on the same Flash-Next files, against Strata and against llama-server:

Engine against engine here, 1.31x at 32k and 1.62x at 100k on decode, is smaller than the agent runs' 1.73-2.05x. The two comparisons don't change the same things: the agent one also changes sampler and context, and measure.py sends prose prompts with thinking off. Neither ratio is clean. The direction agrees, and the weights factor agrees too: Q2_0 over IQ4_XS on Strata is 1.14x at 32k and 1.10x at 100k, with the same expert-cache difference as in the agent runs riding along. MTP alone is worth 1.39x and 1.28x on llama.cpp's decode, and costs it some prefill.

What the logs can't tell#

• llama.cpp flags are reconstructed from git; Strata flags are exact. llama-server logs no command line, so each llama.cpp arm's flags come from my serve script at the bench commit each run recorded. Strata's are exact: they come from the config files the Strata servers ran with (a per-run file in round two).
• Strata's prompt numbers are approximate (whole-second clock), and it logs no cached-token count, so there is no Strata prompt rate for requests that reused a prefix.
• Some Strata requests have no depth. Short prefills print no progress line; for 354 to 679 requests per round-two Strata arm the depth is unknown, and their rates sit outside the depth table.
• Harness side requests count. Titles and background calls land in the same log as agent turns and can't be told apart.

The smaller file did earn something: 3 to 11 % in the agent runs that held the engine build still, 10 to 14 % in the synthetic ones, each with its caveats attached. The rest of the 2.2x goes to the engine bundle. If you're choosing a setup for this box by speed, choose the engine bundle first and the quant second; the quant is the part you can round off.

Filed under: amd, local-llm, ai-agents, benchmarks

Read next#

• Five languages, two reps: the round-two ranking: the same six round-two arms, ranked by failure-path tests and review.
• The best-reviewed code came from the slowest Flash-Next arm: round one's ranking, and the sampler confound this post keeps running into.
• 128k context is the floor, and prefill is the wall: cold prefill at depth, the cost the prefix cache hides in agent runs.
• The benchmark was talking to the previous server: where Strata and its modes first got measured.

Sources#

• Benchmark artifacts (local, not published): per-request rows parsed from each agent run's llama-server or Strata serve log (5,260 requests in round one, 8,933 in round two), run manifests (wall time, build, context), and the serve script and model list at each run's bench commit; 2026-10-12 to 2026-10-30.
• Strata measure.py results for the Flash-Next files, 2026-10-11 to 2026-10-14 (local).
• Strata, tag v0.1.42; issue #4034, on the hipBLASLt tuning table.
• llama.cpp, builds b11538 and b11541, and PR 30105.
• Model repositories on Hugging Face: unsloth/Qwen3.8-Flash-Next-GGUF, ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF, ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF, ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF, IFM/K2-Horizon-MoVA-36B-A4B-GGUF, ornith-ai/Ornith-1.5-35B-A3B-GGUF, unsloth/Qwen3.5-122B-A10B-GGUF, unsloth/North-Mini-Code-1.0-GGUF, unsloth/Qwen3-Coder-Next-GGUF, unsloth/Step-3.7-Flash-GGUF, ggml-org/gpt-oss-120b-GGUF, unsloth/Mistral-Small-4-119B-2603-GGUF.