My benchmark served the model at half speed

Nine short agent tasks, one model, an RTX 3090. llama.cpp finished them in 1,521 seconds. open-jet, a megakernel engine, finished them in 629. That is 2.4 times faster, the kind of ratio that gets a post written about a new engine.

The engine, on a different quant and drafting one token deeper, accounts for 1.29x of it. The rest came from a flag one of my scripts had and the other didn't.

Three arms, one missing flag#

The comparison: Swift-1.5 (a Qwen3.8 27B fine-tune), tasks t10 to t12, three reps each, OpenCode at medium reasoning effort, 64k context and the same client build in every arm. open-jet drafts four tokens per step with the model's multi-token-prediction (MTP) head. Qwen3.8 ships one MTP layer, and llama.cpp can draft with it too: --spec-type draft-mtp. The bench's llama.cpp server wasn't passing it.

Output length doesn't explain it: the median run wrote 5,940 tokens on the baseline and 5,821 on open-jet. Turning MTP on cut llama.cpp's wall by 1.87x. Against llama.cpp with MTP, open-jet is 1.29x faster, still a real margin. (The two engines also load different quants of the same model, which is a separate story.)

How two scripts serving one model drifted#

This repo has two ways to start a GPU server. launch/serve.sh serves the coding clients I use. agent-playground/bench/serve-model.sh serves the benchmark. Both take the same model keys and load the same GGUF files.

On 7 October I measured MTP on five models. The four Qwen3.8 27B builds roughly doubled their decode, from about 43 to 87–96 tokens per second, with 68–73% of drafted tokens accepted. Ornith, a 35B mixture-of-experts model, went from 149 to 205. Within hours the launcher got a per-model default for all five, and launch/serve.sh started doing this:

if [ "$DRAFT" -gt 0 ]; then
  EXTRA_ARR+=(--spec-type draft-mtp --spec-draft-n-max "$DRAFT")
  if [ -n "${MTP_DRAFT[$KEY]:-}" ]; then EXTRA_ARR+=(--model-draft "$HEADFILE"); fi
else
  EXTRA_ARR+=(--spec-type none)
fi

The measurement itself had run on the bench server. It opted in through an explicit BENCH_EXTRA on that one queue (--spec-type draft-mtp --spec-draft-n-max 3 for the Qwen keys). The opt-in did its job, the default went into the launcher, and the bench script kept serving these models the way it always had. From then until the fix on 15 October, every bench row for the four Qwen3.8 keys that didn't ask for drafting ran without it.

No error, no warning. Every one of those runs was scored on its merits, at half the launcher's decode speed.

Why the result rows looked complete#

Each bench row records the server it ran against. This is the serve block of a t16 row written early on 15 October:

"serve":{"engine":"b1-f95b0d9","weights":"Swift-1.5-Qwen3.8-27B-IQ4_XS.gguf","weights_sha256":"e3b59faa312a6202d7ac0a2400123956104bc720e77f6ff21c754aec5c3740df","sampler":{"temperature":1.0,"top_k":20,"top_p":0.949999988079071,"min_p":0.0,"repeat_penalty":1.0,"presence_penalty":0.0,"dry_multiplier":0.0}}

The engine build, the weights' SHA-256, top_p to fifteen significant digits. Whether the server was drafting: not recorded.

Adding it isn't as simple as reading one more key from /props, which is where those fields come from. llama.cpp's /props reports speculative.types as none with MTP running. In the server source, at the build the bench runs, the endpoint's default generation settings come from a freshly constructed parameter struct with only the sampler copied in:

task_params tparams;
tparams.sampling = params.sampling;

Everything else in it keeps its default, and the default speculative type is COMMON_SPECULATIVE_TYPE_NONE (common.h). Current master builds /props the same way. The sampler in /props is the server's real one. The drafting entry next to it describes a struct nobody configured.

/slots is where the truth lives: each slot reports whether it can speculate, plus the parameters of its last task.

The serve log had it all along#

The baseline arm's serve log has zero lines containing draft. Decode on its longer completions (500 tokens or more) held between 36.7 and 39.3 tokens per second, median 38.5. The MTP arm's log opens with creating MTP draft context against the target model and then reports acceptance after every request:

0.14.696.881 I slot print_timing: id  0 | task 0 | draft acceptance = 0.70988 (  115 accepted /   162 generated), mean len =  3.13

Its median decode on the same kind of completions was 76.6 tokens per second. Every bench serve log for these models since 7 October had the same silence, apart from the two queues that opted in: this one and 7 October's measurement. A comparison against an engine that drafts is what got the baseline log opened. The baseline's last row landed at 01:54 on 15 October, open-jet's at 02:05, and the queue adding the llama.cpp-with-MTP arm was written at 02:07.

The fix: BENCH_MTP and a field read from /slots#

serve-model.sh now defaults MTP on for the four Qwen3.8 27B keys, using the launcher's flags. The fifth key in the list, a Q4_K_M build of Swift, was added that night:

case "$LOOKUP" in qwen38-27b-128k|swift15-27b-128k|thinkingcap-27b-128k|qwen38pi-27b-128k|swift15-q4km-128k) _MTPDEF=1 ;; *) _MTPDEF=0 ;; esac
if [ "${BENCH_MTP:-$_MTPDEF}" = 1 ] && [[ " ${BENCH_EXTRA:-} " != *" --spec-type "* ]]; then
  if [ -n "${MTPHEAD[$LOOKUP]:-}" ]; then
    [ -f "$ROOT/models/${MTPHEAD[$LOOKUP]}" ] || { echo "missing MTP head: $ROOT/models/${MTPHEAD[$LOOKUP]}" >&2; exit 1; }
    EXTRA+=(--model-draft "$ROOT/models/${MTPHEAD[$LOOKUP]}")
  fi
  EXTRA+=(--spec-type draft-mtp --spec-draft-n-max 3); echo "BENCH_MTP: on (3 draft tokens)"
fi

BENCH_MTP=0 brings back the old condition for anyone re-running an old comparison. The other half of the fix matters more: every row now carries serve.speculative, read from /slots. Rows on the 27B models record "speculative":"none,draft-mtp". Servers that aren't drafting record "off".

What the old wall times are still worth#

Wall-clock numbers from before the fix for these four models are slow compared with what a launcher user sees. On t10–t12 the gap was the 1.87x above. On t16, a Spring Boot task run three times on each of the four models, the median wall went from 958 s to 596 s, but that pair also moved from 64k to 128k context, so not all of the drop belongs to MTP.

Verdicts are messier. The t10–t12 arms are 9/9 with and without drafting, and both t16 sets came out 3 PASS and 9 PARTIAL. (Since 22 October the scorer no longer counts README docs checks toward the verdict, and that rescore moved two runs of the no-MTP set to PASS: 5 and 7 against 3 and 9, though that pair also changed context.) The Zig pair didn't hold. Swift on the Zig task at 128k, nine runs a side, went from 6 PASS / 3 PARTIAL without MTP to 3 PASS / 5 PARTIAL / 1 FAIL with it, while the median wall fell from 2,167 s to 1,053 s. That pair changed two things at once: the no-MTP runs (11 to 13 October) still carried my ~/.claude prompt into OpenCode, and the MTP runs on 15 October didn't. With three runs per prompt, neither change can be told apart from noise. No claim about MTP and quality follows from any of it, in either direction.

Comparisons inside the old data still hold, because both sides ran under the same handicap. The one-hour walls that four models kept hitting on a Zig task were hit by a server decoding at roughly half the launcher's speed. That post's conclusion compares 64k against 128k on that same server, so it stands. Its absolute times are the slow ones.

A model key shared by two scripts is not a shared configuration. Two launch paths for "the same model" will drift, and a result row only shows the drift if it records the server's effective config, read from the endpoint that actually knows it. Rows from before the fix can't say whether their server drafted. The serve logs can: draft-acceptance lines, or none.

The fix left one out. It covers the four Qwen3.8 keys. The launcher has drafted Ornith by default since the evening of 7 October, and serve-model.sh didn't until I added it on 16 October, while writing this. Every Ornith serve log the bench wrote in between, through 9 October, has zero draft lines.

Filed under: local-llm, llm, benchmarks, llama.cpp

Read next#

• Zig needed context, not time — the wall-clock limits above, from the benchmark side, on the pre-MTP server.

Sources#

• llama.cpp tools/server/server-context.cpp at f95b0d9 — /props builds its default generation settings from a fresh task_params with only sampling copied; slot JSON reports speculative from can_speculate().
• llama.cpp common/common.h at f95b0d9 — default speculative types { COMMON_SPECULATIVE_TYPE_NONE }.
• Bench results, serve logs and scripts are from my local benchmark repository (not public): serve-model.sh, launch/serve.sh, result rows ojbase1/ojmtp1/ojmk1, sb1/sblong1, and their serve logs.