MTP acceptance is a side effect, not a target

Assumes you know what speculative decoding is. Part 6 covers how the bench turned on MTP drafting for the Qwen3.8 27B tunes.

The hard-tier table for Swift-1.5, a Qwen3.8-27B tune, ends in a column that reads 0.83: the share of the MTP head's draft tokens that the model accepted, averaged over every request the server logged. The server drafts three tokens ahead, and the average request keeps 83 percent of them.

The same friend whose 16 GB card started part 15 looked at acceptance figures like that and read them as waste. His advice, in a chat on 20 October: raise the draft maximum from 3 to 4 and keep acceptance around 0.6 to 0.7. Above that, he said, the card sits idle when it could be producing tokens faster. He added that llama.cpp has two settings for this, a max and a min, and that he didn't know how they choose.

It's a testable claim and the GPU was free between two queues, so I swept it.

What the two keys do#

The draft settings live in common/speculative.cpp. In the MTP draft loop, at the build the bench runs (f95b0d953), each step proposes the head's most likely token and then checks two limits:

const llama_token id = cur_p->data[0].id;

// only collect very high-confidence draft tokens
if (cur_p->data[0].p < params.p_min) {
    drafting[seq_id] = false;
    n_drafting--;

    continue;
}

common_sampler_accept(smpl, id, true);

auto & dp = dparams.at(seq_id);
auto & result = *dp.result;

result.push_back(id);

if (params.n_max <= (int) result.size()) {
    drafting[seq_id] = false;
    n_drafting--;
    continue;
}

And after the loop:

if (dp.result->size() < (size_t) params.n_min) {
    dp.result->clear();
}

• --spec-draft-n-max is a cap. Draft until you have N tokens.
• --spec-draft-n-min is not a lower bound on drafting. A draft that comes out shorter than N gets thrown away whole. Default 0.
• --spec-draft-p-min is the one that makes draft length dynamic: the head stops as soon as its top token's probability falls below P. Default 0.0, so out of the box every draft runs to the cap.

The bench runs n-max 3, p-min 0. Every draft but a request's last is exactly three tokens long, so acceptance is a clean ratio here: accepted over three per step.

The sweep: six fixed lengths, four dynamic ones#

One model, one card. Swift-1.5 at IQ4_XS on the RTX 3090 at 280 W, 131,072 context, q8_0 KV cache on the GPU, the model card's sampler (temperature 1.0, top-p 0.95, top-k 20). The arms: MTP off, then fixed n-max 2, 3, 4, 5 and 6, then dynamic length at n-max 4 with p-min 0.7 and n-max 6 with p-min 0.5, 0.7 and 0.85.

Each arm ran three workloads:

• Greedy code. Three prompts (an LRU cache, a CSV invoice aggregator, a topological sort, each with ten unit tests), up to 1,536 output tokens, temperature 0, thinking off. The best case for a draft head.
• Sampled code. The same three prompts with the card sampler and thinking on. That's how the agents run.
• Prose. One request with an 817-token prompt and one with 24,663 tokens.

Results: greedy wanted four, sampling wanted two or three#

Code columns are the mean of three prompts. Acceptance is accepted over drafted tokens across those three requests, from the server log.

Greedy code took the advice and liked it, a little. Four drafts beat three by 3 percent, and acceptance there was 0.80, still above his band. Five and six gave it back.

Sampled code went the other way. Two and three tie at 62.9 and 62.0, inside each other's spread (the three prompts ranged 58.5–71.1 and 58.8–64.3). Four was slower, and six was the worst arm in the sweep that had MTP on at all: 47.4 tokens per second, with individual prompts between 40.2 and 53.8. Prose agreed: two and three tie, longer is worse.

In every column, acceptance fell steadily as the draft got longer. Speed peaked once and fell.

Why acceptance falls while speed peaks early#

The server logs a second number next to acceptance: mean len, which in server-context.cpp is one plus accepted tokens per verification step. That's how many tokens each pass of the full 27B produces. Divide it by tokens per second and you get the time one step takes.

(Step times are derived: tokens per step divided by tokens per second, mean of three prompts. The greedy and sampled figures agree to within 0.3 ms at every length; the table shows greedy's. With MTP off, a token takes 23.8 ms.)

Every extra draft slot adds about 5 to 7 ms to every step, accepted or not. The head runs once per draft token, one after another, and the big model verifies a longer batch. Whether that slot pays depends on one thing: does it add enough tokens per step to cover its cost?

Under greedy decoding, yes, up to four. Tokens per step keep climbing all the way to six, but from five on the extra time costs more than the extra tokens bring. Under the card's temperature-1.0 sampler, the head's guesses run out early. Tokens per step go 2.22, 2.51, 2.71, 2.85, 2.84, so drafts five and six buy nothing at all, and their 7 ms is still charged.

Acceptance is accepted over drafted. Draft one more token that the model mostly rejects and acceptance drops, by construction, whether speed went up or down. It's a ratio of the head's success to its attempts, and nothing in it knows what an attempt cost.

p-min raises acceptance; fixed 3 stays faster#

Dynamic length is the obvious fix: let the head stop when it's unsure. It does what it says. On sampled code, p-min 0.85 at n-max 6 raised acceptance from 0.30 to 0.80, and won back part of what six fixed drafts had lost: 55.0 tokens per second against 47.4, and 49.0 against 40.8 on the long prose prompt. It still didn't catch fixed 3.

On sampled code and prose, no dynamic arm beat fixed 3. On greedy code, two of them edged it by 0.2 and 1.3 tokens per second, inside the prompt-to-prompt spread and still short of fixed 4. Thresholding stops the draft where the head is unsure, which lifts the ratio. Some steps then carry no draft at all, and the server's mean len counts only the steps that verified one, so the step-time arithmetic above doesn't carry over to these arms. Per token, n-max 4 with p-min 0.7 and fixed 3 come out about even on sampled code: 61.0 against 62.0.

The MoE counterexample: fastest well above the band#

The band doesn't transfer to other models either. On 28 September I swept Ornith 1.5, a 35B mixture-of-experts model with an MTP head, at draft lengths 0, 1 and 3, and two days later at 2. Greedy, three coding prompts, 32k context, medians:

The fastest setting runs at 0.89, well above the band. Two drafts land at 0.55, just under it, and give back 23 tokens per second. Three drafts are slower than not drafting at all. The step arithmetic from above explains it. With MTP off, Ornith produces a token every 6.3 ms. One draft slot makes a step 8.8 ms for 1.88 tokens; three make it 13.9 ms for 2.14. That's 2.2 to 2.9 ms per slot on a 6.3 ms token, 35 to 46 percent, against 21 to 30 percent for the dense 27B. (Derived from the logged drafted counts: with a fixed length and p-min 0, steps are drafted tokens divided by the length.)

Where his band came from, and where he could still be right#

Averaged over all eight requests per arm (three greedy, three sampled, two prose), fixed 3 comes out at 0.64. That's inside his band. It's also an average that no single workload has: at fixed 3, greedy code ran at 0.87, sampled code at 0.50 and prose at 0.48. Applied to one workload at a time, the band picks differently. Greedy code reaches 0.70 only at six drafts, which ran at 87.5 tokens per second against 91.9 at four. Sampled code is in the band at two drafts (0.60), tied fastest with three, so on the workload closest to how agents sample, the band says to draft fewer tokens than he suggested. Prose is already below it at two.

The hard-tier logs complicate it the other way. The 0.83 in that table is a mean over 267 requests. Weighted by tokens it's 0.75, and the median request produced 3.67 tokens per step, close to greedy code's 3.60. By tokens per step, agent traffic behaves more like the greedy workload than like sampled code, even though the sampler is the same. I haven't established why. On greedy code, four drafts were 3 percent faster.

So his suggestion to try 4 may hold for real agent runs. I haven't measured it. It's one queue arm, timed on agent traffic.

What this doesn't say#

• One model, one card, one llama.cpp build. Swift-1.5 27B on a 3090 at 280 W. The Ornith numbers come from a different sweep, three weeks older, on an older build.
• Speed only. No agent task was run at each draft length, so there's no pass rate here. MTP is meant to leave the output distribution unchanged, but in practice the greedy outputs weren't byte-identical across arms: at temperature 0 and seed 42 one prompt stopped at 804 tokens with three drafts and ran the full 1,536 with four. The 28 September sweep saw differing greedy hashes too.
• Sampled rows are three requests each, each with different text. Differences of a token or two per second between neighbouring arms are noise.
• The "32k" prose step sends 24,663 tokens, not 32k.

My friend's instinct to try four was worth a sweep, and on greedy code it won. His band would have sent sampled code the other way. Acceptance came out wherever the draft length put it; only the clock picked a winner.

Filed under: local-llm, llm, llama.cpp, benchmarks

Read next#

• My benchmark served the model at half speed — what MTP is worth on this card, and how the bench lost it.
• It fits in 16 GB. That was the easy question — the same friend's first question, and what MTP does with the cache in system RAM.
• The model card said think harder. The audits got worse — another default that the measurement didn't support.

Sources#

• llama.cpp common/speculative.cpp at f95b0d953, MTP draft loop — the p-min stop, the n-max cap and the n-min discard quoted above.
• llama.cpp common/common.h at f95b0d953 — draft defaults: n_max 3, n_min 0, p_min 0.0.
• llama.cpp tools/server/server-context.cpp at f95b0d953 — the draft acceptance log line and the definition of mean len.
• Benchmark artifacts (local, not published): the draft-length sweep's per-request rows, serve-log acceptance lines and VRAM samples (2026-10-21), the 28 September / 30 September MTP sweep rows for Ornith 1.5 and Qwen3.8, and the hard-tier serve log for Swift-1.5.