One open PR made MoE prompts 12-30% faster on my 3090

Assumes you've served a GGUF with llama.cpp. Part 1 introduced the bench.

115 upstream llama.cpp commits bought my RTX 3090 at most 2.8%. The next morning, one pull request from outside upstream bought up to 29.7% in prompt speed in llama-bench, and about 13% through the server that actually feeds the agents, on synthetic prompts. Part 25 moved my llama.cpp build forward by those commits and measured the result on the card: prompt reading moved -0.4% to +2.8%, generation 0.0% to +0.6%. I switched anyway, mostly for the parser fixes.

The card serves local models to coding agents, the subject of this series. The patch is an open pull request in Unsloth's fork of llama.cpp. It changes seven CUDA files and only helps mixture-of-experts (MoE) models, the ones that route each token through a few of many small expert networks. It only speeds up the prompt-reading half of the work, not generation. On two such models it made prompts read 21% to 30% faster in llama-bench at llama.cpp's default batch settings, and 12% to 13% faster through llama-server with the bench's normal serving settings, with the same next-token choices at every position I checked. Even the smaller number is several times what the 115 upstream commits did.

TL;DR — Unsloth's llama.cpp PR #264 (open, CUDA only), merged onto ddf48f85 (the upstream master from part 25) and measured on an RTX 3090 on 2 November:

• Speed: Qwen3.6-35B-A3B read prompts 28.5% to 29.7% faster at the default micro-batch (how many prompt tokens go through the model per pass) of 512 tokens, and 12.7% to 13.4% faster for 2k and 8k prompts at a micro-batch of 2048. Ornith-1.5-35B-A3B gained 21% to 23% at 512. Ornith's generation didn't move, with or without MTP (the model's own draft layer guessing tokens ahead).
• Through the server: llama-server with the bench's serving settings read 2k to 24k prompts 12% to 13.5% faster on both models (one cell, Qwen3.6 at 2k, +8.8%), roughly half the llama-bench gain for Qwen3.6 and three fifths for Ornith. Why the server keeps less, I don't know yet.
• Same outputs: mean KL divergence (how far the two builds' token probabilities differ) against the unpatched build rounds to 0.000000 on both models, and the top token matched at every position. The kernel tests the PR touches pass on the 3090's sm_86, which the PR's own runs didn't cover.
• Method: the first A/B said -26% to +22% for the same model. A rerun with prompt speed only, five repetitions and the two builds alternated gave the clean number.
• Adopted as the default build the same afternoon.

Where the patch came from#

Unsloth publish GGUF quants (the Qwen3.6 file in this test is one of theirs) and, alongside them, prebuilt llama.cpp releases. Each release is an upstream build number plus a list of pull requests merged on top, and the list is printed in the release body. The one from 30 October, b11985-mix-c116c32, names 28 entries: new architectures, new quant types, Vulkan and Metal fixes, nearly all of them pull requests in Unsloth's own fork.

One of them is #264, "CUDA: faster MoE prompt processing (compact MMQ tiling, device expert sort, GET_ROWS)". It changes seven files under ggml/src/ggml-cuda/ plus the backend test file, and nothing else. A change that small and that contained can be tested against my own build without taking anything else from the prebuilt.

So I didn't take the prebuilt. I merged the one pull request onto the master I'd switched to in part 25. It merged cleanly, and the build reports itself as b12024-26-g5f7daf149, built in a CUDA development container like the others. Between the old build and the new one, the only difference is #264.

What #264 changes#

The PR's description lists three changes, without a breakdown of how much each one contributes. The first is the tiling. When a batch of prompt tokens goes through an MoE layer, each token is sent to a few experts, and the matrix multiply runs per expert. Upstream gives every expert its own column tiles, so with many experts and only a few tokens each, most tiles are nearly empty. The PR numbers all experts' tiles consecutively and sizes the tile for an average expert instead of the largest. It also sorts tokens to experts on the GPU instead of the host, and fixes the launch shape of GET_ROWS, the gather kernel that runs around the multiply.

The PR's own numbers come from an RTX PRO 6000 Blackwell and a B200, at a micro-batch of 2048: Qwen3.6-35B-A3B prompts 17% to 19% faster, a dense 8B unchanged. Its "Not covered" list names sm_75 to sm_89 hardware, which compiled but wasn't run. The 3090 is sm_86, so whether the patch computes the same thing on this card was mine to check, along with the speed.

The first A/B said -26% and +22%#

The first speed run kept part 25's settings and order with longer prompts: llama-bench with q8_0 KV and flash attention, prompt lengths of 512, 2,048 and 8,192 tokens at micro-batches of 512 and 2048, plus 128 generated tokens, three repetitions. The micro-batch (-ub) is how many prompt tokens go through the model in one pass; llama.cpp's default is 512. The old build ran first and again last, with the new one in between, so the old build's drift between its two runs shows how big a difference has to be before it means anything.

Ornith came out clean: +21% to +23% at a micro-batch of 512, generation +0.2% and 0.0%, old-build drift 0.8% at most. Qwen3.6, in the same window, came out as nonsense. Its six prompt cells ranged from -26.1% to +22.4%, and the old build disagreed with itself by up to 9.1%: 3,874 tokens per second on its first pass at one setting, 3,538 on its second. With a ruler that bends 9%, a -26% cell says nothing about the patch.

The rerun changed three things. Prompt speed only (-n 0), so no generation pass runs between the prompt tests. Five repetitions instead of three. And the builds alternated, old, new, old, new, old, so each build was measured at three or two points spread across the window instead of in one block:

The old build's three runs now agreed within 0.5%. I never found out what moved the first window, and the rerun didn't repeat it. The alternated run is the number I quote, and alternating is now the default for any A/B on this card.

Biggest gain at the default micro-batch#

At a micro-batch of 512, Qwen3.6 went from about 3,940 to about 5,065 tokens per second on a 512-token prompt, and from about 3,655 to about 4,706 on an 8k prompt. At 2048 the 8k prompt went from about 5,825 to about 6,568. The gain is largest where each pass carries fewest tokens per expert, which is where the PR says upstream's tiles are mostly empty.

That is the case part 25 thought had the least room. There I wrote that a fused MoE kernel, upstream's #31188, should have less to gain at a 512-token batch, because each expert gets few tokens per launch. #264 is a different change, and it gets its biggest gain exactly there, which fits its own description: the nearly empty tiles are what it targets. A 512-token prompt is a single pass at either micro-batch, which is why both columns read nearly the same in the first row.

Most of my serving keys run at the default 512, so that is the column that matters for them, at least in llama-bench; the server, below, keeps less of it. The bigger micro-batch was faster in absolute terms before the patch and still is (6,568 against 4,706 tokens per second on an 8k prompt); the patch narrows that gap from about 1.6x to about 1.4x. Changing the serving micro-batch is a separate knob I haven't tested on those keys.

Through llama-server, half to three fifths of the gain#

llama-bench times the model and nothing else. The agents talk to llama-server, so in the evening the two builds went through it, again old, new, old: the bench's own serving script with its normal settings (q8_0 KV, flash attention, no batch flags, so llama.cpp's defaults) and a fresh server per run. Each request was a fresh string of random words with prompt caching off and one generated token, two requests per size, and the speed is the server's own prompt tokens per second:

The old build agreed with itself within 1.2%, except Ornith at 2k without MTP, at 3.3%. Qwen3.6's 2k cell is the soft one: its two new-build requests landed 6.6% apart. At 24k, Qwen3.6 went from 3,946 to 4,441 tokens per second, and Ornith with MTP on from 3,594 to 4,033. Turning MTP on costs Ornith prefill speed on both builds, about 8% at 24k, 9% at 8k and 13% to 16% at 2k, and the patch's gain comes through either way.

So the server keeps about 12% to 13%, against llama-bench's 28% to 30% for Qwen3.6 and 21% to 23% for Ornith at the same KV and flash-attention settings. I don't know why. The server's log doesn't print the micro-batch it ran, and I haven't chased the gap. One earlier server reading disagrees. The MTP decode probe in the next section also logged prefill for its single 24,663-token request to Ornith with MTP on, and that read +5.4% (3,807.5 and 3,808.3 tokens per second old, 4,015.0 new). That probe used a different prompt, 15 English words repeated in random order with a planted line to recall, and timed the first token from the client side; even the unpatched build read 3,808 there against 3,594 here, so something besides the build moves the speed, most likely the prompt. I don't know which of those differences accounts for the gap. The 12% to 13% comes from repeated requests on one kind of synthetic prompt, and on that kind the number to expect from a served model on this card is the server's, not llama-bench's.

Generation and MTP didn't move#

The patch touches prompt processing only. Ornith's 128-token generation moved +0.2% and 0.0% in the first window. Qwen3.6's generation was measured only in that noisy window, at -2.0% and -1.4%, and the rerun skipped generation, so I have no clean number for it. MTP decoding, where the model's own extra layer drafts tokens for the full model to check, can't be measured in llama-bench, so Ornith ran through llama-server with its one-token draft, a fresh server per build, one request per prompt size, old, new, old:

Decode tokens per second. In part 25, single MTP requests on the same build of Qwen3.8 27B varied by up to about 7% either way. These spread less than that, and the new build sits 1% to 2% below both old runs on the short prompt and between them on the long one. I read that as unchanged, not as a measured loss. Dense models don't use the changed MoE paths, so I didn't benchmark them.

Same outputs, checked before switching#

A faster kernel that changes the answer is a regression. Besides the backend tests and a tool-call check on the two MoE models, a full gate ran in the afternoon, before the patched build replaced anything. It covered the KL divergence, the MTP run above and the tool calls on every serving key:

• Backend tests on the 3090: test-backend-ops for the two operations the PR changes passed, MUL_MAT_ID 984 of 984 and GET_ROWS 340 of 340.
• KL divergence against plain ddf48f85 on both MoE models (details below).
• Tool calls: the two-scenario tool-call probe from part 25, through the bench's serving script, on all nine serving keys. Eight passed. The ninth, Laguna-XS, failed it again, as it has on every build I've run it on.

For the KL divergence check, llama-perplexity saved the old build's token probabilities over 16 chunks of 2,048 tokens of my own project notes, with a q8_0 KV cache, and then compared the new build's probabilities at every position. KL divergence measures how far two probability distributions differ; zero means identical.

The PR reports a maximum of 0.000055 to 0.000061 on its models and calls that equal to running the same build twice. I didn't run that same-build control here, so the comparison is to their floor, not mine.

Adopted the same afternoon#

The gate passed by mid-afternoon, and the patched build became the default on the 3090 box and behind the serving proxy that shares the card. The proxy then sent four live calls through it: a chat, two requests that overflow the context and must come back as clean errors, and one to a 27B model with MTP. None failed. ddf48f85 without the patch stays on disk as the fallback.

What I haven't measured: what 12% to 13% faster server prefill does to a whole agent run. The server's number, not llama-bench's, is the right input; the agent run is still a guess. Part 8 is the warning there: a 2.2x faster prefill finished agent tasks 4% sooner. Most agent turns there reused a cached prompt prefix, which put the ceiling at 1.07x to 1.11x before anything else ate into it. Also not measured: f16 KV cache, and the offload keys that run at a micro-batch of 256.

Part 25 asked whether 115 commits did anything for this card, and the answer was barely. This answer came from a pull request that upstream doesn't have, found in the pin list of somebody else's prebuilt, and its size depended on what was holding the stopwatch: about 29% in llama-bench once the A/B was run properly, about 13% through the server that does the work, on the synthetic prompts I measured it with.

Filed under: local-llm, llm, llama.cpp, benchmarks

Read next#

• 115 llama.cpp commits later, my 3090 was barely faster: the upgrade check this patch went on top of, and the small-batch case it expected least from.
• 2.2x faster prefill, 1.04x faster agent: why a prompt-speed gain shrinks once a coding agent is driving.
• On one RTX 3090, agent speed followed active parameters: where the two 35B MoE models sit on generation speed.
• Image input on a 27B coder costs 1.1 GiB, once: a vision projector next to a 27B coding model; its screenshot and full-context runs used the patched build.

Sources#

• unslothai/llama.cpp PR #264: the patch (open), with its description of the tiling, sort and gather changes, its RTX PRO 6000 and B200 results, KLD figures and "Not covered" list.
• Unsloth llama.cpp prebuilt b11985-mix-c116c32: the release body with the pin list that includes #264.
• llama.cpp compare caa8191...ddf48f85: part 25's 115 commits, the base the patch was merged onto.
• Qwen3.6-35B-A3B GGUF (Unsloth) and Ornith-1.5-35B-A3B GGUF (bartowski): the two MoE models tested.
• My own bench data (local, not published): backend test logs, KLD runs, tool-call probe outputs, both llama-bench windows, the MTP server probe and the llama-server prefill check (old/new/old, two requests per size), 2 November 2026.