Llm on tsvinc

Image input on a 27B coder costs 1.1 GiB, once

Mon, 02 Nov 2026 00:00:00 +0000

Assumes you've served a GGUF with llama.cpp. Part 1 introduced the bench.

I had the memory risk of image input in the wrong place. On a 24 GB RTX 3090 that already holds a 27B coding model with a 128k-token context, I expected the dangerous request to be the full one: an image plus a context filled to the brim. Adding image input to Qwen3.8-27B makes sense: the base model is a vision-language model, and a coding agent that can read a screenshot of a failing UI or a terminal is worth having. The question was whether it still fits on the card.

One open PR made MoE prompts 12-30% faster on my 3090

Mon, 02 Nov 2026 00:00:00 +0000

Assumes you've served a GGUF with llama.cpp. Part 1 introduced the bench.

115 upstream llama.cpp commits bought my RTX 3090 at most 2.8%. The next morning, one pull request from outside upstream bought up to 29.7% in prompt speed in llama-bench, and about 13% through the server that actually feeds the agents, on synthetic prompts. Part 25 moved my llama.cpp build forward by those commits and measured the result on the card: prompt reading moved -0.4% to +2.8%, generation 0.0% to +0.6%. I switched anyway, mostly for the parser fixes.

115 llama.cpp commits later, my 3090 was barely faster

Sun, 01 Nov 2026 00:00:00 +0000

Assumes you've served a GGUF with llama.cpp. Part 1 introduced the bench.

This series is about one 24 GB RTX 3090 and the local models I serve from it as coding agents, each run scored by tests the model never sees. The models are served by llama.cpp, built from source, and since 26 October the build was master at caa8191. By 31 October upstream was 115 commits further on, and the question was the routine one: did any of them do anything for this card and these models?

Faster prompts, slower agents: a second engine on the 680M

Sun, 01 Nov 2026 00:00:00 +0000

Assumes you've run a local model behind a coding agent. Part 1 has the laptop and why every model here is a mixture-of-experts; part 2 has how the agent runs were graded.

TL;DR On a laptop's integrated Radeon 680M, UnbiasedStrata, an inference engine built around mixture-of-experts models (each token runs through a few expert blocks, a small slice of the parameters), read a 2,048-token prompt 1.4 to 1.5 times as fast as llama.cpp's Vulkan backend once warm, but only on models whose experts all fit in GPU memory. It generated slower on every model it ran: 52 to 78% of llama.cpp's speed. Three coding-agent runs on the same model file got the same grades on both engines (2, 3 and 3 out of 5) from part 2's blind review, which grades the code under a random id. They took longer on Strata: one ran out the two-hour clock, one died on the context limit. Agents spend most of their time generating, so llama.cpp stays the engine for agents on this laptop.

The README of UnbiasedStrata closes its human-written note with "Use at your own risk. Works on my machine." My machine is a 2022 laptop with a Ryzen 7 PRO 6850U and its integrated Radeon 680M GPU, which has no memory of its own and borrows about 16 GB of the laptop's 30 GB of RAM. It is not the machine the README means. The engine targets an NVIDIA card with CUDA, and the 680M's chip, gfx1035, appears nowhere in the build's list of AMD GPUs, not even among the ones marked "unvalidated". I read the warning, agreed with it, and built the thing anyway.

105 identical commands and other ways local agents fail

Sat, 31 Oct 2026 00:00:00 +0000

Assumes you've driven a coding agent through a client like OpenCode. Part 1 introduced the bench; part 23 has the results by task.

GLM-4.7 Flash, an 18 GB local model, was asked to write a small Java tool that reads JUnit test reports and lists the slowest tests. Somewhere in the run it typed this shell command:

rm -rf test-reports && mkdir test-reports && java SlowTests --json test-reports 2>&1

The command empties a directory and then asks the tool to summarise it. After a line from the JVM about its options, the tool answered with a JSON summary that began {"total":0. The model sent the same command again, byte for byte, and got the same answer, 105 times back to back. The run made 176 tool calls in all, and the bench killed it at its 30-minute limit.

Most models passed the scripts. The ledgers split them

Sat, 31 Oct 2026 00:00:00 +0000

Assumes you've driven a coding agent through a client like OpenCode. Part 1 introduced the bench; part 22 covered how fast the same models run.

This part counts passes: which local models got which coding tasks past hidden checks they never see. The models run on one RTX 3090 (24 GB of VRAM), mostly served by llama.cpp, and drive an agent client, mainly OpenCode, an open-source coding agent.

TL;DR — Since 15 October, through OpenCode on one RTX 3090, 18 models ran six small command-line tools in their first intake arm (a model's first batch of intake runs under one set of settings): 288 of 309 runs passed every hidden check, and 11 models passed all of theirs. 20 models ran feature work in existing Java, Go and Spring Boot code, again counting each model's first arm: 80 of 180 runs passed, no model passed all 9, five passed none. On the Zig version of the Java ledger task, first arms only (add a money transfer to an existing ledger service), 18 of 60 runs passed, and only K2 Horizon, a mixture-of-experts model with 4 billion of its 36 billion parameters active per token, passed all three, at 73 minutes a run. The best on feature work were tuned variants of the dense Qwen3.8-27B model with MTP drafting, the model's own draft layer (ThinkingCap 8 of 9, Qwen3.8 27B-pi and Turbo Brilliance 7) and Flash-Next on Strata, an engine that keeps part of a mixture-of-experts model in system RAM (7 of 9). A gap of one or two passes out of nine is inside run-to-run noise.

On a laptop iGPU, memory bandwidth picked the model

Sat, 31 Oct 2026 00:00:00 +0000

Assumes you've run a local model behind a coding agent. The 3090 Agent Bench ran the same kind of models on a desktop GPU; this series moves them to a laptop.

I ran local models as coding agents on a laptop: a Ryzen 7 PRO 6850U from 2022, with the models running on its integrated Radeon 680M GPU through llama.cpp. The first request opencode, the coding-agent client I drove them with, sent in a session was 11,000 to 12,400 tokens long, and the model had to read all of it before it wrote a word. A dense 27-billion-parameter model, one that runs every weight for every token, took 299 seconds, about five minutes, to read it. Three mixture-of-experts models of 35 to 40 billion parameters, which route each token through a few expert blocks so that only 2.5 to 3 billion parameters are active, took 77 to 78 seconds. None of them had opened a file of the project yet. Five minutes, and the agent hadn't done a single thing I could blame it for. That part would have to wait as well.

On one RTX 3090, agent speed followed active parameters

Sat, 31 Oct 2026 00:00:00 +0000

Assumes you've run a local model behind a coding agent. Part 1 introduced the bench; this part and the next one sum it up.

The smallest file on my bench is Ternary Bonsai 2, a 27-billion-parameter model packed into 7.2 GB. Driving a coding agent, it generated 37.9 tokens per second, the median over its three batches of runs. Ornith 1.5, a 35B model in a 19.3 GB file, generated 171.2. The file 2.7 times bigger ran 4.5 times faster.

Twenty-one agent runs on a laptop iGPU, none merge-ready

Sat, 31 Oct 2026 00:00:00 +0000

Assumes you've driven a coding agent through a client like opencode. Part 1 covers why every model here is a mixture-of-experts.

TL;DR — Seven mixture-of-experts models (each token runs through only a few expert blocks, a small slice of the parameters) ran three coding tasks each through opencode, a coding-agent client run headless in a sandbox, on a laptop's integrated Radeon 680M GPU: a Node bookmarks API, a Java access-log report and a Go status page. 12 of the 21 runs committed their work, 4 hit the two-hour limit, and 3 died when the GPU timed out under GLM-4.7-Flash, one of the seven. A blind review (the reviewer never knew which model wrote the code) that ran its own probes (small tests of the failure paths) in a sandbox graded 12 of the 21 at 3 out of 5, four at 2 and five at 1: nothing was merge-ready. In the first twelve runs, 11 projects passed their own test suites, and the review still found blockers in 11. Tiel-Coder, a 35B model with about 3B active, came closest.

The third run mattered where it cost the most

Thu, 29 Oct 2026 00:00:00 +0000

Assumes you've driven a coding agent through a client like OpenCode. Part 1 introduced the bench.

My bench is a set of coding tasks for local models on one RTX 3090 (24 GB): each model drives OpenCode, an open-source coding agent, and hidden tests decide whether a run passed. Until this part, a new model went through an intake: every task three times, in three tiers. The basic tier is six small scripting tasks, the hard tier three ledger services in Java, Go and Spring with hidden core tests, and the Zig tier one Zig ledger task that may run for four hours. One model's full intake cost about a day and a half of the card's time, and most of that is Zig.

llama.cpp's MoE cache wants gigabytes and long prompts

Tue, 27 Oct 2026 00:00:00 +0000

Assumes you've served a MoE GGUF with llama.cpp and know what --n-cpu-moe does. The card and the build bump behind these numbers are in The 3090 unlock is real. Where it lands is the catch.

My agent bench runs local models on one RTX 3090 with 24 GB of VRAM. A mixture-of-experts (MoE) model too big for that can keep part of its experts (blocks of weights, of which each token uses only a few) in system RAM. llama.cpp now has a flag for that case, --moe-cache-mib: a slice of VRAM that keeps the popular experts on the GPU while the rest sit in RAM. This part measures what it buys on three MoE models: Flint 40B, Ornith 35B and Flash-Next, the last too big even for VRAM and RAM together.

Flint mistyped its path and blamed the tools

Thu, 22 Oct 2026 00:00:00 +0000

Assumes you've driven a coding agent through a client like OpenCode. Part 1 introduced the bench's sandbox.

On 22 October a Go run of Flint 40B made 115 tool calls, and OpenCode refused 81 of them. The model spent the session working around what its summary called a "persistent tool-output glitch", then reported its code and tests written. The workspace diff was zero bytes.

The glitch was a typo. Seven calls in, the model started mistyping its own workspace path, one token off.

MTP acceptance is a side effect, not a target

Wed, 21 Oct 2026 00:00:00 +0000

Assumes you know what speculative decoding is. Part 6 covers how the bench turned on MTP drafting for the Qwen3.8 27B tunes.

The hard-tier table for Swift-1.5, a Qwen3.8-27B tune, ends in a column that reads 0.83: the share of the MTP head's draft tokens that the model accepted, averaged over every request the server logged. The server drafts three tokens ahead, and the average request keeps 83 percent of them.

Claude Code read my home folder as a project

Tue, 20 Oct 2026 00:00:00 +0000

Builds on My benchmark read my CLAUDE.md.

Part 5 of this series told the story of OpenCode carrying my global ~/.claude into every bench prompt. It held up Claude Code as the client the harness had isolated properly: CLAUDE_CONFIG_DIR pointed at an empty bench-local directory, and claudeMdExcludes kept the repository's CLAUDE.md out.

NoteCorrection to part 5

That isolation didn't hold. Claude Code walks up from its working directory into my home directory and loads my global CLAUDE.md and ~/.claude/rules/ from there, and every bench workspace sat under it. The details are below.

It fits in 16 GB. That was the easy question

Tue, 20 Oct 2026 00:00:00 +0000

Assumes you've run llama.cpp and know what the KV cache is.

"Will it fit?" is the question everyone asks about a model and a graphics card, because it has a clean yes-or-no answer. A friend asked it about a 16 GB card: can Qwen3.8-27B, or its Swift-1.5 tune, at IQ4_XS (13.3 and 14.4 GiB of weights) run fully on the GPU if the KV cache goes to system RAM? He sent the flags he'd run it with, -t 8 -ctk q8_0 -ctv q8_0, and the question was what adding -nkvo does.

The model card said think harder. The audits got worse

Tue, 20 Oct 2026 00:00:00 +0000

Assumes you've run a local reasoning model as a coding agent and know what a reasoning-effort setting does.

The Qwen3.8-27B model card lists three reasoning efforts and marks one as the default:

xhigh (default): for complex tasks demanding thorough analysis

A few lines further down it adds a tip aimed at people like me:

In multi-turn agentic tasks, lower reasoning effort does not always reduce overall task completion time. Although it may produce faster per-turn responses, it can also lead to insufficient analysis, more failures, and repeated retries, which may increase total latency and token consumption.

The hard tier sorted by a rollback, not by speed

Mon, 19 Oct 2026 00:00:00 +0000

Assumes you know how this bench grades: a prompt, a repo, and a hidden suite the agent never sees, where one failing core test makes the run FAIL.

On 19 October this post had a title, "Fast models failed the hard tier", and three runs per model to back it. The two fastest decoders on the box had just gone 0 of 9. So had Muse Glimmer, whose label in my model notes read "slow; never failed on scripting or dev tools", although the same row already recorded a failed Java ledger in all four earlier arms, each for one missing rollback. I parked the draft until the remaining queues finished, so it would not age into "still running".

The hidden test asked for more than the prompt

Mon, 19 Oct 2026 00:00:00 +0000

Assumes you've seen a hidden-test agent benchmark: the model gets a task prompt and a repo, a scorer runs tests the model never saw.

When most models fail the same check, the check starts to look like the hard part of the task. It sits in the results as a wall: four tunes of one 27B model hit it, six other models hit it, run after run. The natural reading is that the models are missing something.

The harness decided how long the model thought

Sat, 17 Oct 2026 00:00:00 +0000

Assumes you've run a coding agent against a local llama.cpp server and know what prefill, decode and reasoning tokens are.

On the first run of t11, a Bash script that files an inbox of messy files by what they contain, Qwen3.8-27B wrote 62,977 characters of reasoning. The run took 425 seconds and passed. On the first run of the same task in Pi, against the same server settings, it wrote 8,911 characters, took 90 seconds, and passed.

2.2x faster prefill, 1.04x faster agent