llama.cpp
- Image input on a 27B coder costs 1.1 GiB, once
Image input for Qwen3.8-27B coding keys on an RTX 3090 cost 1.1 GiB at load on every variant. A full 121k-token context with a screenshot added 118 MiB.
- One open PR made MoE prompts 12-30% faster on my 3090
115 llama.cpp commits gave my RTX 3090 2.8% at most. One open PR from Unsloth's fork made MoE prompts 21-30% faster in llama-bench, 12-13% in llama-server.
- The screenshot opencode never sent
Local vision models on a Radeon 680M read test screenshots almost perfectly. Three opencode versions sent an error string instead until one config key was set.
- 115 llama.cpp commits later, my 3090 was barely faster
I checked 115 new llama.cpp commits on an RTX 3090: tool calls held, speed moved 2.8% at most, adaptive MTP gave nothing. I switched anyway.
- Faster prompts, slower agents: a second engine on the 680M
A MoE engine read prompts 1.4x faster than llama.cpp on a Radeon 680M and generated slower. Three agent runs got the same grades and finished later.
- On one RTX 3090, agent speed followed active parameters
Generation speed of 28 local models driving coding agents on one RTX 3090. Active parameters and MTP drafting set it; the quant moved it far less.
- On a laptop iGPU, memory bandwidth picked the model
Coding agents on a Radeon 680M: a dense 27B generated 5.7 tokens a second whatever I tried; MoEs with 2-3B active on the GPU, 27 to 48.
- llama.cpp's MoE cache wants gigabytes and long prompts
On a 24 GB RTX 3090, llama.cpp's MoE expert cache read long prompts up to 85% faster, did nothing for a Q8 model, and left a 2-4x gap to Strata.
- The 3090 unlock is real. Where it lands is the catch
gpu_unlock lifted my RTX 3090's BF16 GEMM from 74.9 to 128 TFLOPS. My daily 27B in llama.cpp got 0.7% faster at short context. Here's where the rest went.
- The proxy said unloaded. The model was loading.
A context probe gave up on four models loading behind my LLM proxy: its status was a snapshot up to 30 s old. Then a 336 ms stream timed at 120 s.