Llama.cpp on tsvinc

Image input on a 27B coder costs 1.1 GiB, once

Mon, 02 Nov 2026 00:00:00 +0000

Assumes you've served a GGUF with llama.cpp. Part 1 introduced the bench.

I had the memory risk of image input in the wrong place. On a 24 GB RTX 3090 that already holds a 27B coding model with a 128k-token context, I expected the dangerous request to be the full one: an image plus a context filled to the brim. Adding image input to Qwen3.8-27B makes sense: the base model is a vision-language model, and a coding agent that can read a screenshot of a failing UI or a terminal is worth having. The question was whether it still fits on the card.

One open PR made MoE prompts 12-30% faster on my 3090

Mon, 02 Nov 2026 00:00:00 +0000

Assumes you've served a GGUF with llama.cpp. Part 1 introduced the bench.

115 upstream llama.cpp commits bought my RTX 3090 at most 2.8%. The next morning, one pull request from outside upstream bought up to 29.7% in prompt speed in llama-bench, and about 13% through the server that actually feeds the agents, on synthetic prompts. Part 25 moved my llama.cpp build forward by those commits and measured the result on the card: prompt reading moved -0.4% to +2.8%, generation 0.0% to +0.6%. I switched anyway, mostly for the parser fixes.

The screenshot opencode never sent

Mon, 02 Nov 2026 00:00:00 +0000

Assumes you've run a local model behind a coding agent. Part 1 has the laptop and its memory pool; the router manual has the serving setup.

TL;DR On a laptop's integrated Radeon 680M, llama.cpp served 14 model presets (12 models, two of them in two quants) with image input, each with its projector, the small extra file that turns pixels into tokens. On five generated test images they got 54 of 56 single answers right, and read a six-line cut list back with full recall on every model. Mixture-of-experts models with spare GPU memory lost 2% of their generation speed or less; dense 27B models and the presets that already filled the pool lost 4 to 32%. In opencode, a terminal coding agent, versions 1.19.3 and 1.19.5 on my setup and 1.18.35 on a second machine did not send an attached image unless the model entry declared image input: the model got an error string instead, and the run exited 0. Adding "modalities": {"input": ["text", "image"], "output": ["text"]} fixed it. One setup that first saw the image arrive without the key had it in a config file in a parent directory, which opencode merges in. An RTX 3090 desktop measured the same projector memory with no speed loss. On the second machine, a Strix Halo mini PC, two models "passed" the image test without seeing the image by running an OCR tool through bash, so check what was sent and what tools ran, not the answer. On 18 questions about real screenshots with thinking on, Gemma 4 26B (a mixture-of-experts model) got 17 at 23 s each on average, and Swift 9B (a dense model) all 18 at 38 s.

The obvious suspect for anything new on this laptop is the GPU. It's a Radeon 680M with no memory of its own: on the boot these tests ran on, it borrowed a 4 GiB carve-out plus 13.6 GiB of system RAM it can map (GTT), out of the laptop's 32 GB, and this series has already spent a fair share of its words on what does and doesn't fit in it. So the place to expect trouble from letting local coding models look at screenshots is right there: a picture means extra weights, extra tokens and extra memory, on a GPU that has none to spare.

115 llama.cpp commits later, my 3090 was barely faster

Sun, 01 Nov 2026 00:00:00 +0000

Assumes you've served a GGUF with llama.cpp. Part 1 introduced the bench.

This series is about one 24 GB RTX 3090 and the local models I serve from it as coding agents, each run scored by tests the model never sees. The models are served by llama.cpp, built from source, and since 26 October the build was master at caa8191. By 31 October upstream was 115 commits further on, and the question was the routine one: did any of them do anything for this card and these models?

Faster prompts, slower agents: a second engine on the 680M

Sun, 01 Nov 2026 00:00:00 +0000

Assumes you've run a local model behind a coding agent. Part 1 has the laptop and why every model here is a mixture-of-experts; part 2 has how the agent runs were graded.

TL;DR On a laptop's integrated Radeon 680M, UnbiasedStrata, an inference engine built around mixture-of-experts models (each token runs through a few expert blocks, a small slice of the parameters), read a 2,048-token prompt 1.4 to 1.5 times as fast as llama.cpp's Vulkan backend once warm, but only on models whose experts all fit in GPU memory. It generated slower on every model it ran: 52 to 78% of llama.cpp's speed. Three coding-agent runs on the same model file got the same grades on both engines (2, 3 and 3 out of 5) from part 2's blind review, which grades the code under a random id. They took longer on Strata: one ran out the two-hour clock, one died on the context limit. Agents spend most of their time generating, so llama.cpp stays the engine for agents on this laptop.

The README of UnbiasedStrata closes its human-written note with "Use at your own risk. Works on my machine." My machine is a 2022 laptop with a Ryzen 7 PRO 6850U and its integrated Radeon 680M GPU, which has no memory of its own and borrows about 16 GB of the laptop's 30 GB of RAM. It is not the machine the README means. The engine targets an NVIDIA card with CUDA, and the 680M's chip, gfx1035, appears nowhere in the build's list of AMD GPUs, not even among the ones marked "unvalidated". I read the warning, agreed with it, and built the thing anyway.

On a laptop iGPU, memory bandwidth picked the model

Sat, 31 Oct 2026 00:00:00 +0000

Assumes you've run a local model behind a coding agent. The 3090 Agent Bench ran the same kind of models on a desktop GPU; this series moves them to a laptop.

I ran local models as coding agents on a laptop: a Ryzen 7 PRO 6850U from 2022, with the models running on its integrated Radeon 680M GPU through llama.cpp. The first request opencode, the coding-agent client I drove them with, sent in a session was 11,000 to 12,400 tokens long, and the model had to read all of it before it wrote a word. A dense 27-billion-parameter model, one that runs every weight for every token, took 299 seconds, about five minutes, to read it. Three mixture-of-experts models of 35 to 40 billion parameters, which route each token through a few expert blocks so that only 2.5 to 3 billion parameters are active, took 77 to 78 seconds. None of them had opened a file of the project yet. Five minutes, and the agent hadn't done a single thing I could blame it for. That part would have to wait as well.

On one RTX 3090, agent speed followed active parameters

Sat, 31 Oct 2026 00:00:00 +0000

Assumes you've run a local model behind a coding agent. Part 1 introduced the bench; this part and the next one sum it up.

The smallest file on my bench is Ternary Bonsai 2, a 27-billion-parameter model packed into 7.2 GB. Driving a coding agent, it generated 37.9 tokens per second, the median over its three batches of runs. Ornith 1.5, a 35B model in a 19.3 GB file, generated 171.2. The file 2.7 times bigger ran 4.5 times faster.

llama.cpp's MoE cache wants gigabytes and long prompts

Tue, 27 Oct 2026 00:00:00 +0000

Assumes you've served a MoE GGUF with llama.cpp and know what --n-cpu-moe does. The card and the build bump behind these numbers are in The 3090 unlock is real. Where it lands is the catch.

My agent bench runs local models on one RTX 3090 with 24 GB of VRAM. A mixture-of-experts (MoE) model too big for that can keep part of its experts (blocks of weights, of which each token uses only a few) in system RAM. llama.cpp now has a flag for that case, --moe-cache-mib: a slice of VRAM that keeps the popular experts on the GPU while the rest sit in RAM. This part measures what it buys on three MoE models: Flint 40B, Ornith 35B and Flash-Next, the last too big even for VRAM and RAM together.

The 3090 unlock is real. Where it lands is the catch

Mon, 26 Oct 2026 00:00:00 +0000

NVIDIA sells the GeForce RTX 3090 with one kind of tensor math running at half speed: FP16 and BF16 multiplies that accumulate in FP32, and TF32, issue at half the rate of the professional cards built on the same silicon. gpu_unlock, an open-source kernel module, puts the full rate back at every boot. My 3090 runs headless at home as a local LLM box, mostly serving one 27-billion-parameter coding model through llama.cpp, so the question was what doubling that math buys the work I actually do. I approach every free speedup the same way: I assume the bill is somewhere, and I go looking for it.

The proxy said unloaded. The model was loading.

Sun, 25 Oct 2026 00:00:00 +0000

Ten times in sixteen seconds the probe asked my proxy for its model list, and ten times it got back the same 54,031 bytes, with the model it had just asked to load listed as unloaded. After fifteen seconds of that, the probe wrote its verdict and moved on:

{"alias": "3090-flint-40b", "configured_context": 131072, "result": "error", "error": "3090-flint-40b returned to unloaded during load", "total_s": 18.4}

Four of the seven models in that run got the same line. None of them had unloaded. The 3090 box's router had accepted each load, exactly as asked.

"Terminating immediately" took 90 seconds

Sat, 24 Oct 2026 00:00:00 +0000

llama-server keeps a message for the moment you run out of patience with it: Received second interrupt, terminating immediately. On the 3090 box it printed that at 09:45:04. The next thing systemd had to say about the unit came at 09:46:34, and it was Aborting.

That stop was half of an engine switch. My LAN proxy for local models stops one llama.cpp build and starts another when a request names a model that only the other build has (why it does that). The proxy logged the new engine up after 1m36s. A switch between those two builds takes 2 to 4 seconds. Two signals did this, the second one came from my own script, and the "immediately" path deadlocked inside glibc.

Cutting a stream doesn't cancel the turn

Sat, 24 Oct 2026 00:00:00 +0000

The plan was a router. One OpenAI- and Anthropic-compatible URL on a mini-PC, two GPU boxes behind it, read the request's model, pick the box, forward the bytes. I started it at 23:37 one night. By 22:04 the next evening it was starting and stopping llama.cpp builds on the 3090 box (the A9's benchmarks kept it out that day), deciding when a request has to wait, and shutting engines down after fifteen idle minutes so a benchmark could have the GPU back.

To a coding agent, 503 means try again

Sat, 24 Oct 2026 00:00:00 +0000

opencode run printed its banner, > build · mock-err503, and then nothing. No error, no exit code, no output, until the run was killed. The mock server behind it answered every request with a 503 in my LAN proxy's error format: the status the proxy sent at the time for any model it couldn't serve, whether the GPU box was down, the engine switch had failed or a benchmark owned the GPU for the next few hours. opencode was retrying, and it had no plans to stop.

MTP acceptance is a side effect, not a target

Wed, 21 Oct 2026 00:00:00 +0000

Assumes you know what speculative decoding is. Part 6 covers how the bench turned on MTP drafting for the Qwen3.8 27B tunes.

The hard-tier table for Swift-1.5, a Qwen3.8-27B tune, ends in a column that reads 0.83: the share of the MTP head's draft tokens that the model accepted, averaged over every request the server logged. The server drafts three tokens ahead, and the average request keeps 83 percent of them.

It fits in 16 GB. That was the easy question

Tue, 20 Oct 2026 00:00:00 +0000

Assumes you've run llama.cpp and know what the KV cache is.

"Will it fit?" is the question everyone asks about a model and a graphics card, because it has a clean yes-or-no answer. A friend asked it about a 16 GB card: can Qwen3.8-27B, or its Swift-1.5 tune, at IQ4_XS (13.3 and 14.4 GiB of weights) run fully on the GPU if the KV cache goes to system RAM? He sent the flags he'd run it with, -t 8 -ctk q8_0 -ctv q8_0, and the question was what adding -nkvo does.

My benchmark served the model at half speed

Fri, 16 Oct 2026 00:00:00 +0000

Nine short agent tasks, one model, an RTX 3090. llama.cpp finished them in 1,521 seconds. open-jet, a megakernel engine, finished them in 629. That is 2.4 times faster, the kind of ratio that gets a post written about a new engine.

The engine, on a different quant and drafting one token deeper, accounts for 1.29x of it. The rest came from a flag one of my scripts had and the other didn't.