llama.cpp's MoE cache wants gigabytes and long prompts

Assumes you've served a MoE GGUF with llama.cpp and know what --n-cpu-moe does. The card and the build bump behind these numbers are in The 3090 unlock is real. Where it lands is the catch.

My agent bench runs local models on one RTX 3090 with 24 GB of VRAM. A mixture-of-experts (MoE) model too big for that can keep part of its experts (blocks of weights, of which each token uses only a few) in system RAM. llama.cpp now has a flag for that case, --moe-cache-mib: a slice of VRAM that keeps the popular experts on the GPU while the rest sit in RAM. This part measures what it buys on three MoE models: Flint 40B, Ornith 35B and Flash-Next, the last too big even for VRAM and RAM together.

50.6 tokens/s without the cache, 30.6 with it. That was Ornith 35B as a 35.2 GiB Q8 file, with a large share of its experts in RAM and a 2 GiB cache. Without the cache, the CPU computes those RAM experts while the model generates. With it, a hit runs on the GPU and a miss first copies the expert over PCIe. A cache that misses a lot should cost a little. Forty percent is not a little.

Most of that 40% belonged to my setup. What's left after taking it out is the cache's real profile on a 24 GB card: worth gigabytes of VRAM in one situation, a trade in another, and roughly nothing in the rest.

TL;DR — Measured on one RTX 3090 (24 GB), one run per cell (one setup at one prompt length), llama.cpp build caa8191 (master, 26 October):

• Model fits on the GPU: the cache has nothing to do; per the source, it only caches layers whose experts are all in RAM.
• Many experts in RAM, long prompts: prompt reading got up to 85% faster on Flint 40B (a MoE with 2.5 billion of its 40 billion parameters active per token) at a ~77k-token prompt, with a 3 GiB cache, and its generation gained up to 24%.
• Few experts in RAM, small cache: 512 MiB sped up Flint's long-prompt reading by 16% and slowed its short-prompt generation by 16%. Keeping more of the model on the GPU still gave the best short-prompt generation.
• A quant that doesn't fit versus one that does: Ornith 35B as a 35.2 GiB Q8 file with experts in RAM was 2–3x slower to generate and 4–5x slower to read prompts than its 19.3 GB IQ4_XS file entirely on the GPU (best Q8 setup, prompt chunk size -ub matched).
• A model bigger than VRAM plus RAM: Strata, an engine built for this one model family that keeps its most-used experts in a GPU cache, read prompts 3.3–3.9x faster than llama.cpp on the same 62 GiB of Flash-Next weights (a 2-bit MoE). Generation was 2.0–2.6x faster, but Strata drafts with MTP (the model's own extra layer proposing tokens for the full model to check) and llama.cpp didn't.
• Three setup mistakes changed the answer: a forced -ngl 99 switches --fit off, memory-mapped experts under a memory limit, with the short prompt measured cold, made the cache look 40% worse, and a smaller -ub cost one side 12–23% of its prompt speed.

What --moe-cache-mib caches, and when it only reads#

The flag arrived in PR #29887, merged on 7 October and shipped in v0.7.0. On 19 October PR #30637 replaced its plain LRU with a 50/50 segmented LRU, which its author measured at "about 1-5% better than simple LRU". That second version is the one I tested.

The header describes the job in one line: it "keeps recently and repeatedly used experts of host-resident MoE layers in a device buffer". A few comments in src/llama-moe-cache.cpp set the limits:

• Which layers: "only cache layers that keep all of their experts in host memory". A layer with even some of its experts on the GPU isn't cached, so with everything on the card there's nothing for the flag to do.
• Eviction: "probation and protected LRU lists; hits promote slots to the protected half". A newly cached expert starts in probation; a hit moves it to the protected half, and new arrivals evict "from probation first", so a burst of one-off experts doesn't flush the regulars.
• Prompts: "large batches use most experts of a layer, so they gain little from the cache and would evict the experts used in generation". Large means more than 32 tokens (max_batch = 32), and prompts are read in chunks far bigger than that, so prompt reading doesn't fill the cache: "large batches only read the cache, so the experts used in generation stay in it".

That last rule decides where the cache can pay off. Generation fills it. As I read the code, prompt reading takes the experts that generation left in the slots straight from VRAM and copies the rest over PCIe, as it would without a cache. In the Flint runs the short prompt went first, so the long prompts found a cache already warmed by earlier answers.

How I measured: one card, three models, one run per cell#

The box is the bench's: an RTX 3090 at 280 W, unlocked as in the unlock post, a 6-core Ryzen 5 5600G and 31 GiB of DDR4. llama.cpp is master at caa8191, from 26 October.

My measurement client sends one streaming request per depth: filler text with a hidden fact a third of the way in, then up to 1200 generated tokens. Prompt speed is prompt tokens divided by time to the first token, generation speed is the rest of the reply divided by its time. The script calls its depths 1k, 32k and 100k. The prompts that arrived were 817, 24,663 and 76,971 tokens (16 fewer each for Flint's tokenizer), so the tables below say ~1k, ~25k and ~77k.

Every cell is one run. I treat differences under about 10% as noise, and the 77k "needle" check says whether the model found the hidden fact.

-ngl 99 switched --fit off before anything ran#

The first attempt never measured anything. The plan was to let --fit choose how many of Flint's experts go to RAM, with and without a cache. Five arms, five failed loads, and every log said the same thing:

0.00.410.109 W common_fit_params: failed to fit params to free device memory: n_gpu_layers already set by user to 99, abort
0.03.596.900 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 1904.00 MiB on device 0: cudaMalloc failed: out of memory
0.03.597.454 E llama_init_from_model: failed to initialize the context: failed to allocate buffer for kv cache

The bench's serve script passes -ngl 99 to every model. common/fit.cpp refuses to second-guess that: if n_gpu_layers differs from the default, it throws "n_gpu_layers already set by user". It refuses a user-set tensor split or tensor overrides the same way. Note the level on that first line: W. The "abort" ends the fitting, and the server carries on with 99 layers on the GPU, trying to fit a 128k KV cache next to all of Flint's weights. Three seconds later it runs out of memory, with an error that says nothing about --fit. I rewrote the Flint test with explicit -ncmoe values, and started the later tests' servers directly, without -ngl 99.

Flint 40B: the cache paid off where the experts were in RAM and the prompt was long#

Flint 40B-A2.5B, UD-Q4_K_XL, 128k context, q8_0 KV. -ncmoe N keeps the experts of the first N layers in RAM. The bench runs Flint at N=8, which is what fits 128k on the card. Raising N frees VRAM for a bigger cache: the N=12 and N=16 cache arms spend part of what their N freed, and the N=8 arm squeezes 512 MiB into what's left at N=8.

VRAM after load grew by each cache's size (+510, +1536, +3072 MiB), so the slots were really allocated.

Prompt reading at depth is where the cache earned its VRAM. With the same N, the ~77k prompt read 16%, 45% and 85% faster, and the gain grew with the number of layers in RAM. At ~1k, where there's little to read, nothing moved.

Generation depended on the cache's size. The 3 GiB cache at N=16 sped it up by 11% at ~1k and 24% at ~25k. The 512 MiB cache at N=8 slowed the ~1k reply from 100.3 to 84.4 tokens/s. A still-open PR #30848 says small caches are a known problem: it warns about caches "too small to hold a single token", and ramps the protected half from 0% at one token's worth of experts to 50% at 1.8x. I didn't compute Flint's per-token expert size, so whether 512 MiB falls under that line here is a guess I didn't measure.

At similar VRAM, the cache beat more experts on the GPU at depth. N=12 without a cache used 21,030 MiB; N=16 with 3 GiB used 21,990, about 1 GB more. The cache arm read the ~77k prompt at 1952 tokens/s against 1313 and generated at ~25k at 85.0 against 73.8. At ~1k it read 17% slower.

Moving experts to RAM to make room for a cache cost the short prompts. Against the bench's N=8 without a cache, the N=12 and N=16 cache arms read the ~1k prompt 19% and 33% slower and generated about 10% slower there. They won the ~77k prompt by 10% and 13% (1905 and 1952 against 1726), and ~25k generation by 6–8%, which is inside my noise band. The best long-prompt reader of all six arms was N=8 with the small cache, at 2005, the same arm that lost 16% of its short-prompt generation. If the model almost fits, keep it as close to fitting as you can, and put any cache on top of that.

The needle at ~77k was missed in all six arms, with and without a cache, so the miss is the model's.

Ornith Q8: the 40% that was mostly my setup#

The second question was the one a cache seems made for. Ornith-1.5-35B-A3B runs on the bench as IQ4_XS, 19.3 GB, entirely on the GPU. The official Q8_0 is 35.2 GiB, so --fit leaves a large share of its experts in RAM. Could Q8 plus a cache get close to IQ4?

Both Q8 arms ran --fit on --fit-target 256 -ngl auto -ub 256, 128k context, q8_0 KV, MTP off. The first run:

Generation 28–40% slower with the cache, the ~1k prompt down 64%. Taken at face value, the cache is a regression. Before believing that, I listed what else in the run could produce it, and found three things:

1. The experts in RAM were memory-mapped. llama.cpp said so at load, in a warning I hadn't acted on: tensor overrides to CPU are used with mmap enabled - consider using --load-mode none for better performance.
2. The file didn't fit the box. 35.2 GiB of weights on a machine with 31 GiB of RAM, run in a systemd unit with MemoryMax=28G, and that limit also counts page cache. Mapped experts could be dropped and read back from disk mid-run.
3. The short prompt ran first. The ~1k cell would have paid for any cold page-ins, and it's the worst cell in the table.

The rerun changed all three: --load-mode none, no memory limit, and the depths in the order ~25k, ~77k, ~1k.

Without mmap, Q8 read prompts 1.4–1.9x faster than before. The cache's 40% loss shrank to 10% at ~1k and ~77k, a 3% gain at ~25k, and prompt reading about 10% lower: neutral to slightly negative, at the edge of my noise band. The cache was never the headline. My setup was.

--fit pays for the cache out of the weights. The PR that added the cache says so: with --moe-cache-mib, "--fit counts the cache and keeps fewer whole expert layers on the GPU". Here a 2048 MiB cache raised the loaded VRAM by only 150 MiB (23,696 to 23,846 MiB with mmap). So the cache spends VRAM that resident experts would otherwise have used, and it can only pay if it picks experts better than --fit's fixed placement did.

IQ4 on the GPU won by a distance. The IQ4_XS arm first ran with the default -ub 512, the Q8 arms with -ub 256, which they needed for room. Chunk size moves prompt speed, so I reran IQ4 at 256:

The smaller chunk costs IQ4 12–23% of its prompt speed and nothing measurable in generation. With chunks matched, IQ4 on the GPU read prompts 4.0–4.9x faster than the best Q8 setup and generated 2.0–3.0x faster. Without the match, its prompt lead would have read as up to 6.4x. This is speed only: I didn't compare the two quants' answers, and the needle was found in every Ornith cell.

Flash-Next Q2_0: same weights on llama.cpp and Strata#

The third model doesn't fit anywhere. Qwen3.8-Flash-Next GSQ-RCO Q2_0 is two GGUF files, 61.9 GiB together, against 24 GB of VRAM and 31 GiB of RAM. llama.cpp ran it with --fit on --fit-target 256 -ngl auto -ub 256, q8_0 KV, memory-mapped (the file is twice the size of RAM), no MTP. Strata, at commit 1678de3, is an engine built for this one model family. It keeps the experts it uses most in a GPU cache and the rest in page-locked RAM, and here ran with int8 KV and MTP drafting on.

Prompt reading: 3.3–3.9x for Strata, and that comparison is fair. Drafting doesn't touch the prompt.

Generation: 2.0–2.6x for Strata, and that comparison is not. Strata drafts with an MTP head, llama.cpp ran without one. The fair arm would be Strata with MTP off, and this pack won't start that way:

strata generate: /data/packs/q2_0 is a native (IQ) pack: it needs --native SHARD1, --spec T (T >= 2) and --prefill CHUNK

Strata's generation also barely moved with depth, 86.8 tokens/s at ~1k and 80.5 at ~77k, while llama.cpp's fell from 43.9 to 30.9. Its log after the last request shows how rarely it had to look past the GPU:

strata serve: decode expert cache hit rate: 98.4% (224575 hits / 228321 lookups)
strata serve: resident RAM: 18.93 GiB of experts in RAM, 2214 exchanged with the VRAM tier, 0 blob reads from the file

llama.cpp's 4 GiB cache was a mixed bag again: generation +18% at ~1k and +13% at ~77k, −9% at ~25k, prompt reading 4–20% lower. None of it changes the comparison.

Part 1 measured the same weights on both engines on 8 October, and the gap was much wider then: llama.cpp decoded at about 10 tokens/s with --n-cpu-moe 22. Between those runs the llama.cpp build, the expert placement, the power cap, the card's unlock, the measuring script and Strata's sampler (greedy then, as part 1's correction notes; temperature 1.0 from its config now) all changed, so I can't tell you which of them closed most of the distance. I can tell you it was closed by something other than the cache.

Strata can't take Ornith off llama.cpp's hands either. Same vendor, different architecture: the GGUF headers say Ornith is qwen35moe and Flash-Next is qwen4exp, with twice as many experts per layer and per-layer input embeddings. Strata's engine and packer handle only the second.

Two more PRs whose gains didn't reach this card#

Same build, same week. Two open llama.cpp PRs reported decode gains for a q8_0 KV cache, the KV type every llama.cpp arm above ran. PR #30792 reads q8_0 KV directly in the CUDA flash-attention tile kernel; its author measured +19.0% decode at 16k and +61.9% at 63k, on an AMD gfx1151 under ROCm. PR #31073 keeps the KV rotation matrices on the GPU; +8.70% decode on an RTX 3090 attached to an old laptop over PCIe 2.0.

I applied each to caa8191, and both together (#31073 needed a hand merge), then ran llama-bench on Qwen3.8-27B IQ4_XS with q8_0 KV at depths 0, 32k and 100k. Every result landed within −1.0% to +0.3% of the average of two unpatched runs, and those two unpatched runs differed from each other by up to 2.9%. On this box both patches are invisible.

What follows is a reading; I didn't measure it. The architecture doesn't explain it: Qwen3.8-27B keeps attention, and so a KV cache, in one layer in four (part 15 ran into the same detail), and #30792's test model is the same kind of hybrid. The rest of the setup differs. That model is a 35B-A3B MoE, which reads about 3B parameters of weights per token, so its KV reads are a bigger share of the work than in a dense 27B. It ran on an AMD gfx1151 under ROCm. And #31073's gain came from a PCIe 2.0 link, far slower than this card's, so there was more transfer time to save.

When to turn on --moe-cache-mib#

• Everything fits on the GPU: leave it off. It caches nothing.
• A few layers' experts in RAM: don't move more to RAM to make room for a cache. Flint's N=12 and N=16 cache arms read short prompts 19–33% slower than N=8.
• Many experts in RAM and long prompts: try it, with gigabytes. At N=16, 3 GiB bought Flint 85% at a ~77k prompt; at N=8, 512 MiB cost it 16% of generation at a short one.
• Experts in RAM at all: load them with --load-mode none, and don't measure under a memory limit the file doesn't fit in.
• Using --fit: remember the cache comes out of what --fit would have kept on the GPU, and make sure nothing upstream passes -ngl.
• Comparing arms: match -ub, and run the short prompt last or twice.

Filed under: local-llm, llm, llama.cpp, benchmarks

Read next#

• The 3090 unlock is real. Where it lands is the catch: the card and the llama.cpp build these numbers ran on.
• I blamed OpenCode; a 2-bit model passed Zig in it: where Flash-Next and Strata first met this box, and the MTP caveat in full.
• It fits in 16 GB. That was the easy question: the other half of a MoE that won't fit, the KV cache in system RAM.
• The engine got credit for the quant: another engine comparison where one variable turned out to be two.

Sources#

• llama.cpp PR #29887: adds the GPU cache for MoE experts kept in host memory (merged 2026-10-07).
• llama.cpp PR #30637: replaces the LRU with a 50/50 SLRU (merged 2026-10-19), "about 1-5% better than simple LRU".
• llama.cpp PR #30848: open; warning and protected-share ramp for small caches.
• src/llama-moe-cache.cpp at caa8191: the comments quoted above (the header line is in llama-moe-cache.h).
• common/fit.cpp at caa8191: the n_gpu_layers already set by user abort.
• llama.cpp PR #30792: open; native q8_0 KV in the CUDA flash-attention tile kernel, with the author's gfx1151 numbers.
• llama.cpp PR #31073: open; KV rotation constants kept on the GPU, with the author's PCIe 2.0 numbers.
• Strata: the engine used for the Flash-Next comparison.