Mistral-Small-4 ran one matmul on the CPU, and said nothing

· 9 min read
Contents

A slow local model has an obvious diagnosis: it is too big for the box. Drop to a smaller quant, or a smaller model, and move on.

On the A9 Mega that diagnosis was wrong twice in one afternoon. Mistral-Small-4 generated 3.4 tokens per second, with the model loaded on the GPU and llama-bench sitting at about 300 % CPU. MiniMax-M2.7 held the GPU at 100 % busy for hours and drew about 40 W doing it.

Neither model was too big. Mistral asks for FP32 precision on one matmul per layer, no GPU backend in llama.cpp implements that, and the scheduler runs it on the CPU. A pull request that is still open takes decode from 3.4 to 40.1. MiniMax runs into a flash-attention kernel that gets slower with every token already in the cache; switching backends fixed its prefill and broke its decode.

Mistral-Small-4 at 3.4 tok/s, against an estimate of 40#

Mistral-Small-4-119B at UD-Q4_K_XL is a mixture of experts: 128 experts, 4+1 active per token. Only a few billion parameters work on any one token. The note written when its speed screen was cut put the expectation at about 40 tok/s. That was an estimate, written before anyone knew the cause.

llama.cpp b11538, Vulkan, flash attention on, pp512 / tg128:

Mistral-Small-4 UD-Q4_K_XLd032k
b11538 (79e2e74eb)351 / 3.4161 / 3.2

Prefill looked healthy. Generation was a twelfth of the estimate, and %CPU for llama-bench hovered around 300 while it ran. A GPU benchmark that keeps three CPU cores busy is running part of the model on the CPU.

The output had none. The screen's stderr is ten lines: the same two-line Vulkan device banner, five times. No warning, no fallback notice, nothing about where any op ran.

GGML_PREC_F32 on the expert down-projection#

The cause is in build_moe_ffn, in src/llama-graph.cpp, at our build's tag:

experts = build_lora_mm_id(down_exps, cur, selected_experts, down_exps_s, slots); // [n_embd, n_expert_used, n_tokens]
if (arch == LLM_ARCH_MISTRAL4) {
    // src1 can exceed F16 range
    ggml_prec_set_src(experts, GGML_PREC_F32, 1);
}

For Mistral 4 only, the expert down-projection asks for its activations in FP32, because they can overflow FP16. That projection is a MUL_MAT_ID, and no GPU backend supports MUL_MAT_ID at that precision. The scheduler does what a scheduler does with an op the GPU refuses: it hands it to the CPU. In every layer, for every token, and the benchmark carries on.

This isn't a bug in the usual sense. The answer is right. It arrives at 3.4 tok/s, and nothing tells you why.

Upstream: one issue, two pull requests, all open#

As of 13 October:

  • #30224 reports the same fallback on CUDA. Its title has the line that matters: the expert down-projections fall back to CPU "despite 37/37 layers offloaded".
  • #30260 fixes it for CUDA only. Its description says so: "HIP and MUSA retain their existing behavior." So ROCm is no way out.
  • #30105 fixes it for Vulkan, keeping FP32 MUL_MAT_ID on the GPU. Its author reports around 10× on a Radeon AI Pro 9700. That's the author's card and the author's number.

PR #30105 on the A9: 3.4 to 40.1#

Cherry-picking the PR onto b11538 failed, because the PR branch contains a merge commit. So the test build is the PR head itself, a05ddca: master from 7 October plus one fix commit.

Mistral-Small-4 UD-Q4_K_XL, pp512 / tg128d032k
b11538 (79e2e74eb)351 / 3.4161 / 3.2
PR #30105 head (a05ddca)258 / 40.1136 / 24.0

Decode is about 12× faster at d0 and 7.5× at 32k. The estimate of about 40, written that afternoon before the PR run, landed on 40.1.

Prefill paid for it: about a quarter slower at d0 (351 to 258) and about a sixth at 32k (161 to 136). The FP32 path is now on the GPU, and it costs there too.

Checking the fast build against the slow one#

A 12× speedup from a precision change deserves suspicion. The output check ran greedy prompts on both builds, and the slow one makes a good reference: its CPU path does the down-projection in full FP32.

Primes, Go code and all eight needle answers came back byte-identical. The 350-word echo differed: each build dropped a single word, a different one.

Under the first scorer both builds failed that echo, 9 of 10 each. It compared words by position, so one skipped word shifted every word after it: 0.34 for the slow build, 0.29 for the fast one. An aligned ratio (Python's difflib) scores both answers 0.9986, and with it every check passes on both. The models had been fine. The ruler hadn't.

MiniMax-M2.7: prefill gone by 32k#

The second slow model failed differently. MiniMax-M2.7 at UD-IQ3_XXS reads a prompt at a reasonable speed into an empty context, and its prefill falls off a cliff from there. A diagnostic run, pp512 tok/s:

depthVulkan, FA onVulkan, FA offROCm (kyuz0 rocm-10.0)
0304285—
8k151161—
16k65.976.7226.9
32k11.445.2170.0

At 32k, turning flash attention off is about 4× faster, and ROCm is about 15× faster than the default. When the same weights run that much faster on another kernel, the weights are not the problem.

GGML_VK_PERF_LOGGER=1 names the op#

llama.cpp's Vulkan backend has a per-op timer: set GGML_VK_PERF_LOGGER=1 and every graph prints the time and throughput of each op. Plotted against the KV length already in the cache:

Two lines from the log, first and last of that slide:

FLASH_ATTN_EXT dst(128,48,512,1),  q(128,512,48,1),  k(128,512,8,1),  v(128,512,8,1),  m(512,512,1,1): 62 x 337.207 us = 20906.9 us (19105.3 GFLOPS/s)
FLASH_ATTN_EXT dst(128,48,512,1),  q(128,512,48,1),  k(128,31744,8,1),  v(128,31744,8,1),  m(31744,512,1,1): 62 x 602317 us = 3.73436e+07 us (663.159 GFLOPS/s)

The shapes are the model: 48 query heads, 8 KV heads, head dimension 128. The 62 x is the layer count. MiniMax runs full attention in all 62 layers, so every layer pays the falling rate, and there is no cheap layer to hide behind. At d0 the picture is the other way round: attention runs near 19 TFLOPS and the expert matmuls, at 4 to 5, are what cap prefill near 300.

So "Vulkan beats ROCm on this chip" depends on the model. It held for gpt-oss decode in the earlier rows. A full-attention model at long context goes the other way.

100 % busy at about 40 W#

The speed screen went further before anyone looked at the logger. Its 128k row started at 11:57 and was still running at 15:39, 3.7 hours later, with roughly six hours estimated to go. The rule for a model already out of contention is to cut any row that would take over an hour, keep the finished rows and write a .note. The row was killed.

For those 3.7 hours the telemetry said the GPU was 100 % busy. Package power over the row had a median of 39.5 W.

"Busy" means a kernel is resident on the GPU, nothing more. An attention kernel that had already dropped below one TFLOPS by 31k keeps the counter pinned at 100 for hours. The power meter is the instrument that noticed.

ROCm rescues the prefill, and decode collapses#

ROCm was the obvious escape, and for prefill it worked. On the kyuz0 rocm-10.0 image, pp512 / tg128 ran 320 / 29.9 at d0, 164 / 7.1 at 32k and 106 / 3.5 at 64k. Against the Vulkan speed screen's 13.5 and 4.7, that's 12× and 23× more prefill.

Then look at the second number. Decode falls from 29.9 to 3.5 by 64k, where Vulkan had held 36.0, 15.6 and 10.0 at the same depths. A model that reads a prompt fast and then answers at 3.5 tok/s is out either way. MiniMax is out on both backends.

The OOM killer's first two picks#

The ROCm screen ended at 128k, at 06:05 on 12 October, and not by rule:

2026-10-12T06:05:12+03:00 a9-mega kernel: Out of memory: Killed process 64909 (dbus-broker-lau) total-vm:9640kB, anon-rss:124kB, file-rss:24kB, shmem-rss:4kB, UID:1000 pgtables:56kB oom_score_adj:200
2026-10-12T06:05:12+03:00 a9-mega kernel: Out of memory: Killed process 64911 (dbus-broker) total-vm:5504kB, anon-rss:204kB, file-rss:8kB, shmem-rss:0kB, UID:1000 pgtables:52kB oom_score_adj:200
2026-10-12T06:05:12+03:00 a9-mega kernel: Out of memory: Killed process 99620 (llama-bench) total-vm:149609772kB, anon-rss:16735764kB, file-rss:12kB, shmem-rss:693160kB, UID:1000 pgtables:51000kB oom_score_adj:0

The OOM killer took a process holding 124 kB of anonymous memory, then one holding 204 kB, both the user session's message bus, and only then the 16 GiB process that caused the shortage. The oom_score_adj column explains the order: the two bus processes carried 200, llama-bench carried 0. dbus-broker restarted; the bench stayed dead.

The arithmetic says the row never fit. The model is 74.6 GiB. f16 KV for 131072 tokens is 62 layers × 8 KV heads × 128 × 2 (K and V) × 2 bytes = 248 KiB per token, so 31.0 GiB. llama-bench under ROCm with -lm none held 16.0 GiB of anonymous memory on top. That's 121.6 GiB before compute buffers, the OS and page cache, on a box where MemTotal is 124.9 GiB.

Every bench and server process now starts with oom_score_adj 1000 (choom on the host, podman --oom-score-adj in containers), so the process that causes the shortage is the first candidate. An OOM-killed depth ends the screen with its finished rows kept.

A checklist for the next slow model#

  • Sweep depth, then try a second backend. A model slow at 32k but not at d0 is an attention problem, and the backend switch says whose.
  • Run the per-op logger. GGML_VK_PERF_LOGGER=1 turns "slow" into one op name and a TFLOPS figure.
  • Watch %CPU of the bench process. A GPU benchmark above 100 % is running something on the CPU, whatever the loader says.
  • Compare watts to busy. Flash-Next's prefill-bound job in part 5 drew 91 W at 98.9 % busy. 100 % busy at 40 W is a slow kernel holding the GPU, not a saturated GPU.
  • Spend two minutes on a diagnostic run before a long queue. llama.cpp replaced -mmp/--mmap with -lm/--load-mode. The queued ROCm jobs still passed -mmp 0. The diagnostic run's ROCm leg died on it first, in its third line of output: error: invalid parameter for argument: -mmp.

On master, as of 13 October, Mistral-Small-4's down-projection still asks for FP32. Until #30105 or something like it lands, the scheduler will keep honouring the request without a word.

Filed under: amd, linux, local-llm, benchmarks, vulkan

Sources#