benchmarks
- 2.2x faster prefill, 1.04x faster agent
A new engine prefilled 2.1-2.2x faster than llama.cpp at every depth. Driving a coding agent, it finished 4% sooner. The server logs say where the speed went.
- The engine got credit for the quant
open-jet beat llama.cpp 7 to 3 on a Zig agent task. On open-jet's own GGUF, llama.cpp went 7 of 9 too. Only the speed gap belonged to the engine.
- My benchmark served the model at half speed
The launcher turned on MTP drafting for five models. The benchmark's own server script didn't, and its result rows had no field that could show it.
- My benchmark read my CLAUDE.md
An audit built to check sampler overrides found my OpenCode bench runs carrying my global ~/.claude/CLAUDE.md and skills: 53,483 characters of prompt.
- The best-reviewed code came from the slowest Flash-Next arm
Stage 1 of the A9 agent study: the slowest Flash-Next arm reviewed best, the 62 GiB Q2_0 passed 6/6 at nearly 6x the rate. A sampler confound splits them.
- Zig needed context, not time
Four 27B local models kept hitting the one-hour wall on a Zig task. Four hours at 64k got 7 of 12 through the core tests. At 128k, 35 of 36 did, cleanly.
- IOMMU off buys a 128 GB iGPU 5 %, and costs the NPU
On this Strix Halo box the GPU was never behind IOMMU translation, so iommu=pt changed nothing for it. amd_iommu=off bought ~5 % prefill and killed the NPU.
- The benchmark was talking to the previous server
A Flash-Next-only engine, three modes, and a mode result that was really the previous run's server: how cd && server & made $! lie.
- Mistral-Small-4 ran one matmul on the CPU, and said nothing
Mistral-Small-4 decoded at 3.4 tok/s with the model on the GPU: one op per layer ran on the CPU, silently. MiniMax was slow in a different way.
- Vulkan or ROCm on Strix Halo? Ask at which depth
On an empty context Vulkan won the first-day bench on every model. At 127k, ROCm prefilled gpt-oss-120b 7× faster. The answer depends on depth.