115 llama.cpp commits later, my 3090 was barely faster

Assumes you've served a GGUF with llama.cpp. Part 1 introduced the bench.

This series is about one 24 GB RTX 3090 and the local models I serve from it as coding agents, each run scored by tests the model never sees. The models are served by llama.cpp, built from source, and since 26 October the build was master at caa8191. By 31 October upstream was 115 commits further on, and the question was the routine one: did any of them do anything for this card and these models?

A routine check should be routine: build the new master, run it, compare, repoint. This one took two builds and under 20 minutes of GPU time, and the speed answer was close to nothing. On the way, two things almost gave me a wrong answer. A prompt cache made one model read a 24,663-token prompt at 39,338 tokens per second, and the new binary turned out to look for its own libraries in a directory that only exists inside the container that built it.

TL;DR — llama.cpp master ddf48f85 (build 12024) against caa8191 on one RTX 3090, measured on 1 November:

• Tool calls: a two-scenario probe passed on 8 of 9 models; the ninth, Laguna-XS, failed the same way it did on both older builds.
• Speed: prompt reading moved -0.4% to +2.8%, generation 0.0% to +0.6%, against the old build measured before and after the new one. Only one mixture-of-experts (MoE) model's prompt gains beat the old build's drift between its own two runs, which reached 1.3%.
• Adaptive MTP (open PR #27210, which raises or lowers how many tokens the model's MTP layer drafts ahead, depending on how many of its recent drafts were accepted): no measurable gain over a fixed draft of 3 tokens on a 27B, on a prose prompt.
• I switched anyway, kept the old build as the fallback, and flagged bench rows from 1 November as a new build condition.
• Two traps: the server's prompt cache made a repeated prompt read about 33 times faster, and the binary's runpath (where it looks for its own libraries) points at a path inside the build container, so an ldd check passes or fails depending on the directory you run it from.

What the 115 commits brought#

Most of the 115 commits, caa8191..ddf48f85, don't touch anything I serve. The ones that do, for a roster that is mostly Qwen3.5-family models and two 35B MoE models:

• Chat parser fixes for the Qwen3-Coder format, the parser llama.cpp uses for most of my models: #30557 stops reasoning and content before the template's separators, and #31380 stops json_schema requests failing with a 400 when reasoning isn't split out of the reply (reasoning_format: none). Parser changes cut both ways: the first moves where a reply's content ends before a tool call, and a change there can fix tool calls or break them.
• A fused MoE kernel, #31188: the gate and up expert matmuls plus the GLU after them run as one launch during prompt processing. Upstream's table shows +5.7% to +6.1% on an RTX 5090 and +9.1% to +9.9% on a DGX Spark for Qwen3.6-35B-A3B, one of my two MoE models.
• MTP rollback fixes, #31350 and #31368, for recurrent state when drafted tokens get rejected; upstream reproduced the bug with MTP plus a second drafter, not MTP alone as I run it. MTP is the model's own extra layer drafting tokens for the full model to check; my Qwen3.8 27B keys draft three at a time.

One open PR was worth a build of its own: #27210, adaptive MTP draft depth. Instead of a fixed three drafted tokens, --spec-type draft-mtp-adaptive moves the depth up after a run of fully accepted drafts and down after misses; the author suggests a ceiling of 12.

The PR no longer applied cleanly. Master had moved the memory-fit code into a new function, common_fit_apply(), and the PR still edited the old inline copy, to make it treat the adaptive type as MTP. I kept master's refactor and moved the PR's one-line intent, a params.speculative.has_mtp() check, into the new function. The PR's own unit test passed. Both builds, plain master and master plus the PR, took about 15 minutes together at nice 19.

Tool calls before speed#

A faster build that garbles tool calls is a regression, so the first step on the GPU was a tool-call probe, not a benchmark. It has two scenarios, each sent streaming and non-streaming: a plain call to a get_time tool, and a write_file call whose content must be exactly 123, true, null, [1, 2] or {"a": 1}, typed values that a parser can coerce or a model can decorate.

Nine serving keys ran it on the new master: six built on Qwen3.8 27B and its tunes, the two 35B MoE models, and Laguna-XS 2.1. Eight passed. Laguna-XS failed on one of its ten write_file calls:

S2 val='true'     stream=False finish=tool_calls content='this text, nothing else: true' type=str COERCED/WRONG

The prompt asks for content set to "exactly this text, nothing else: true", and the model copied the tail of the instruction into the file. The streaming version of the same request came back as true. It failed the same way on caa8191 and on f95b0d9 before it, so this is the model, not the parser: on this probe, the two parser PRs broke nothing on my roster.

Old build, new build, old build again#

The speed run used llama-bench on three models: Qwen3.6-35B-A3B (UD-Q4_K_M) and Ornith-1.5-35B-A3B (IQ4_XS), both MoE, and the dense Qwen3.8-27B (IQ4_XS). Each ran pp512 and tg128, at an empty context and at a 32k-token depth, q8_0 KV cache, flash attention on, three repetitions.

The old build ran first and again last, with the new one in between. One measurement before an upgrade compares two different card states. Two of them, bracketing the new one, show how far the old build drifts by itself, and anything smaller than that drift isn't the upgrade. Here it drifted up to 1.3%. New build against the mean of the two old runs:

Generation at 32k moved 0.0% to +0.5%. In absolute terms, Qwen3.6 read 512-token prompts at 3,828 to 3,877 tokens per second across the three runs and generated about 153 to 154; the dense 27B generated 42.3 to 42.4 at an empty context on both builds.

Ornith's +2.8% and +2.3% are the only results well above the worst drift, and above Ornith's own drift (0.7% and 0.6%). The 32k figure is the sturdier one: at pp512, the spread within each run of three was about 2%, close to the gain itself. Qwen3.6's +1.4% at 32k clears the worst drift by a tenth of a point. The gains are on the two MoE models, and the dense model moved nothing, which is where a MoE-only kernel would show up. At this size I wouldn't credit #31188 with them.

Why upstream's +6% didn't show up here#

Upstream measured #31188 with llama-bench -p 16384 -b 2048 -ub 2048 -ctk f16 -ctv f16: a 16k-token prompt, a 2048-token micro-batch and an f16 KV cache. Mine was a 512-token prompt at the default micro-batch with a q8_0 cache, on a different card. At pp512 the whole prompt is a single 512-token batch, so each expert gets few tokens per launch, and I'd expect a fused launch to have less to gain there. I didn't test the big-batch case, so all I can say is that at pp512 it's worth at most 3% here, not that upstream's numbers are wrong for their setup.

MTP speed only shows through the server#

llama-bench has no speculative decoding, so it can't measure MTP at all. For that, the Qwen3.8 27B ran through llama-server with its usual draft of three: one streamed request at a time with thinking off, at two depths, twice each. The "32k" request held 24,663 prompt tokens and the "1k" one 817, and each reply ran 357 to 512 tokens. Again the old build went first and last:

The same old build, same model, same requests, decoded 58.6 at 1k in the first arm and 66.6 in the last one. At 1k its two arms sat 8 tokens per second apart, more than any other build's gap from it. At 32k the adaptive arm's two requests read 50.4 and 57.1. With single requests that noisy, master's 55.6 and 56.8 at 32k against the old build's 52.7 to 54.3 is two numbers each, not a gain I'd claim.

Adaptive depth has a reason to show nothing on this test. The probe asks for about 300 words of prose, and the PR's own description calls a depth of 2 or 3 close to optimal for reasoning and prose, and makes climbing past 3 deliberately hard. I didn't read the acceptance rate for these requests, so I can't say whether the depth ever left 3. What the run shows is narrower: on this card, this model and a prose prompt, adaptive depth bought nothing.

Trap one: 24,663 tokens in 0.63 seconds#

The first repetition of the 32k request took 20.4 seconds before the first token came back, which works out to about 1,206 prompt tokens per second. The second repetition, same arm, came back in 0.63 seconds: 39,338 tokens per second. Every arm did it, old build included: rep 2 at 32k read 39,338 to 40,351.

The cause is in the probe's first lines:

rnd = random.Random(7)
WORDS = "ledger account transfer balance mutex thread lock deposit withdraw audit clock store money key replay".split()
def filler(ntok):  # ~1.3 tokens per word for this vocabulary; calibrated by usage below
    return " ".join(rnd.choice(WORDS) for _ in range(int(ntok / 1.3)))

Each repetition starts the script again, so the generator restarts from seed 7 and builds the same filler and the same hidden "vault code" sentence as the run before it. The server stays up between the two repetitions, and llama-server keeps a prompt cache in host memory: earlier prompts stay there after the slot has moved on to other work. When rep 2's 32k request arrived, the server's only slot held the 1k request that had just run, and the server evaluated only 4 of the 24,663 tokens. The rest came back from that cache. Rep 2 at 1k was a cache hit too; its prefill just happened to look normal.

The decode numbers are unaffected, because generation starts after the prompt either way. All rep-2 prefill numbers are thrown out. Clearing the slot between repetitions wouldn't fix it, since the slot isn't where the hit came from. What would: vary the filler per repetition, start the server with --cache-ram 0, or send "cache_prompt": false with each request. It's the same mechanism that makes agent turns cheap, which part 8 went through: here it just turned a speed probe into a cache probe.

Why I switched anyway#

The numbers say neutral, and I switched. Even a small gain is worth taking when it costs nothing and doesn't break anything, and this one had passed the tool-call probe on the whole roster. The parser fixes matter more to me than the speed: they touch the format most of my models speak, and I'd rather have them before a model trips over the old behaviour. A build close to upstream also keeps the next upgrade a small jump instead of a few hundred commits.

On 1 November, three things moved to ddf48f85: the serving proxy that shares the card with the bench, the local launch defaults and the bench defaults. caa8191 stays on disk as the fallback. Bench rows from 1 November carry a flag for the new build, so nothing compares across the switch without saying so. The adaptive-MTP build isn't used.

After the switch, a check through the proxy's own service: engine up 1.3 seconds after start, replies stamped b12024-ddf48f85c, 147 tokens per second on a short chat with a MoE key and 69 on the Qwen3.8 key with MTP. One error text changed: an over-long request now gets request (80011 tokens) exceeds the available context size (65536 tokens), try increasing it, and the proxy's rewrite of that message for Anthropic-API clients still matched. No rollback.

Trap two: a runpath from inside the build container#

While repointing, an ldd of the new llama-server without LD_LIBRARY_PATH reported libllama-server-impl.so => not found. The runpath, the list of directories baked into the binary for finding its libraries, explains it:

$ readelf -d llama-server | grep RUNPATH
 0x000000000000001d (RUNPATH)            Library runpath: [/src/llama.cpp/build/bin:]

/src/llama.cpp/build/bin is where the libraries were built, inside the rootless container that compiled llama.cpp. It doesn't exist on the host. The old build has the identical line, and it has served since 26 October without complaint, because the serving unit sets LD_LIBRARY_PATH to the build directory, and with that every library (ggml, ggml-cuda, the CUDA 13 runtime and cuBLAS) resolves from there and the driver's libcuda from the host.

The trailing colon adds a second catch. It leaves an empty entry in the list, and on this host the loader treated it as the current directory. The same check gives two answers depending on where it runs:

$ ldd ./llama-server | grep server-impl
	libllama-server-impl.so (0x00007fadcf600000)
$ cd / && ldd /path/to/build/llama-server | grep server-impl
	libllama-server-impl.so => not found

Run from inside the build directory, it passes. Run from anywhere else, it fails. A library check that passes in exactly the directory where you'd normally run it isn't checking much. The fix, which I haven't applied yet, is a runpath of $ORIGIN, the binary's own directory: set at configure time with -DCMAKE_BUILD_RPATH_USE_ORIGIN=ON, or patched afterwards with patchelf --set-rpath '$ORIGIN'. (CMAKE_INSTALL_RPATH only applies to cmake --install, and my build copies the binaries straight out of build/bin.) Until then the environment variable does the job, and the binary depends on it.

Next time#

The next upgrade check gets the same order: build, tool-call probe on every serving key, then speed with the old build on both sides. Three things change. The speed probe gets a different filler for each repetition, so no request can be served from the cache, the server log of every arm is kept so the MTP acceptance rate can be read, and the library check runs from /.

Filed under: local-llm, llm, llama.cpp, benchmarks

Read next#

• MTP acceptance is a side effect, not a target: the draft-length sweep on Swift-1.5, a tune of the same Qwen3.8 27B, that adaptive depth was meant to beat.
• The 3090 unlock is real. Where it lands is the catch: the last build switch, f95b0d9 to caa8191, measured on the unlocked card.
• 2.2x faster prefill, 1.04x faster agent: the cached prefix again, this time doing its job for agent turns.
• On one RTX 3090, agent speed followed active parameters: what sets generation speed across the whole roster.

Sources#

• llama.cpp compare caa8191...ddf48f85: the 115 commits.
• llama.cpp PR #31188: fused MoE gate/up matmuls and GLU in MMQ (merged 2026-10-28), with upstream's bench command and RTX 5090 / DGX Spark table.
• llama.cpp PR #27210: adaptive MTP draft depth (open), with the suggested --spec-draft-n-max 12 and the note on depth for prose.
• llama.cpp PR #30557 and #31380: Qwen3-Coder parser fixes (merged 2026-10-28 and 2026-10-31).
• llama.cpp PR #31350 and #31368: MTP rollback fixes (merged 2026-10-29 and 2026-10-30).
• My own bench data (local, not published): the tool-call probe outputs, three llama-bench tables per build, the llama-server speed probe results and the A/B script, 31 October to 1 November 2026; the readelf/ldd checks run on 1 November.