Faster prompts, slower agents: a second engine on the 680M

2026-11-01 · 12 min read

Contents

Assumes you've run a local model behind a coding agent. Part 1 has the laptop and why every model here is a mixture-of-experts; part 2 has how the agent runs were graded.

TL;DR On a laptop's integrated Radeon 680M, UnbiasedStrata, an inference engine built around mixture-of-experts models (each token runs through a few expert blocks, a small slice of the parameters), read a 2,048-token prompt 1.4 to 1.5 times as fast as llama.cpp's Vulkan backend once warm, but only on models whose experts all fit in GPU memory. It generated slower on every model it ran: 52 to 78% of llama.cpp's speed. Three coding-agent runs on the same model file got the same grades on both engines (2, 3 and 3 out of 5) from part 2's blind review, which grades the code under a random id. They took longer on Strata: one ran out the two-hour clock, one died on the context limit. Agents spend most of their time generating, so llama.cpp stays the engine for agents on this laptop.

The README of UnbiasedStrata closes its human-written note with "Use at your own risk. Works on my machine." My machine is a 2022 laptop with a Ryzen 7 PRO 6850U and its integrated Radeon 680M GPU, which has no memory of its own and borrows about 16 GB of the laptop's 30 GB of RAM. It is not the machine the README means. The engine targets an NVIDIA card with CUDA, and the 680M's chip, gfx1035, appears nowhere in the build's list of AMD GPUs, not even among the ones marked "unvalidated". I read the warning, agreed with it, and built the thing anyway.

The reason was a number. Parts 1 and 2 ran every model through llama.cpp on Vulkan, and a separate test on another machine, a Strix Halo-class mini PC (AMD's Ryzen AI Max line of chips) running this engine through HIP, AMD's GPU compute interface, had put it at 1.3 to 2 times llama.cpp Vulkan. A speedup measured on another machine is a rumour with a decimal point, so the question for this part is plain: on a 680M, does it make coding agents faster? I didn't reproduce that 1.3 to 2x here, and that test's setup differed from mine.

UnbiasedStrata keeps the experts in a GPU cache#

A mixture-of-experts (MoE) model routes each token through a few small expert blocks out of many, so a 35B model reads only about 3B parameters per token. The whole file still has to sit somewhere. UnbiasedStrata's main idea is a cache of experts in GPU memory, with the rest served from system RAM and swapped in as they get used.

It comes from a short family tree: Strata, then PolyStrata (v0.1.45-poly.1), then UnbiasedStrata, which I ran at commit 22abfdb and call Strata for short from here on. It keeps PolyStrata's two engines unchanged and replaces everything around them: no installer, no downloads at build or start, and a C++ front end, unbiased-strata, that serves an OpenAI-style chat API from llama.cpp's own server code. The README's human-written note gives its reasons in two of its bullets, "piping instructions from markdown into agent == bad idea" and "replacing components at build-time or runtime isn't great", and I've rarely agreed with a bullet list faster.

The engine I used is strata-poly, which reads a GGUF file as it is and accepts four model architectures: qwen35moe, glm5next, gemma4 and lfm2moe. Everything after that note, including the family tree and the correctness work, sits under its own heading, "AI generate nonsense from here on out", so I read it with that label attached. It describes teacher-forced comparisons against llama.cpp, with the engine's per-step token probabilities about as close to llama.cpp's GPU run as llama.cpp's own CPU run is (mean KL divergence 0.6 to 1.2 times that spread).

Building it for gfx1035 took one word and 17 GB#

HIP is an opt-in backend. The build file validates gfx1100 and gfx1201, accepts two community-tested chips, lists a set of other AMD chips as "unvalidated" (RDNA2's gfx1030, gfx1031, gfx1032 and gfx1034 among them) and stops with a FATAL_ERROR for anything else. gfx1035 is anything else. The one change I made to the source was adding that one word to the "unvalidated" list in cmake/hip_backend.cmake: a one-word edit to a list whose whole job was to say no.

ROCm, the AMD compute stack HIP runs on, came from AMD's TheRock Python wheels for the gfx103X family, version 7.13.0, installed into a Python virtual environment, no root needed. That was 14 GB installed plus 3 GB of pip cache I had to purge afterwards: 17 GB of disk to set up a GPU that has about 16 GB to work with. /dev/kfd, the device ROCm talks to, was readable without root, and rocminfo saw "gfx1035" with 16 compute units.

The rest was quiet. The pinned llama.cpp submodule meant the build fetched nothing else, and strata-poly plus the front end built in 4.5 minutes. The binaries have no RPATH, so at run time LD_LIBRARY_PATH has to point at the wheel's library directories.

One precaution I didn't skip: the 680M also draws the laptop's screen. Earlier the same morning GLM-4.7-Flash on llama.cpp Vulkan had hit 6 GPU compute ring timeouts, the same failure that ended its runs in part 2. So every Strata run went into a systemd unit with a memory cap and a hard timeout, and after each one I checked the kernel log for GPU queue timeouts. Strata hit none, in any run.

Strata produced the same tokens as llama.cpp, up to a near tie#

Before timing anything, I checked that the port wasn't broken. Both engines got the same raw prompt token ids, "The capital of France is Paris. The capital of Germany is Berlin. The capital of Italy is", with greedy decoding, on Nail: a fine-tune of Qwen3.6-35B-A3B, in the UD-IQ3_S quant, 14.3 GiB.

The first 16 generated tokens were identical on both: " Rome. The capital of Spain is Madrid. The capital of". At the next step the two split on a near tie: Strata put 0.30 on " the" and 0.27 on the token llama.cpp chose. Strata gave identical tokens on both repeated runs. I read it as the same model behaving the same way, not a broken port.

In server mode, a chat request with a bash tool came back as a proper tool_calls entry, with the reasoning split off. One difference turned up there. opencode, the coding-agent client from parts 1 and 2, sends a grammar that constrains how the model may write a tool call, and llama.cpp enforces it. Strata ignores it, and says so itself: "this engine does not constrain sampling, parsing only". I'd like more tools to confess their limits in one line like that.

Prompt reading was faster only when every expert fit the GPU#

The speed test was a 2,048-token prompt followed by 128 generated tokens. llama.cpp ran it as llama-bench with two repetitions; Strata ran once or twice per model, and where it ran twice the second run is the warm one.

model (GGUF, size)	Strata prompt t/s	llama.cpp prompt t/s	Strata gen t/s	llama.cpp gen t/s
Nail UD-IQ3_S, 14.3 GiB	270	189	14.3	26.4
plain Qwen3.6-35B-A3B UD-IQ3_XXS, 12.3 GiB	192 cold, 279 warm	191	21.9	28.1
Tiel-Coder-35B-A3B UD-IQ4_XS, 16.9 GiB*	106 cold, 96 warm	203	9.7 cold, 13.7 warm	18.8
Groot-19B-A3B Q4_K_M, 11.6 GiB	refuses to load	292	-	24.7

* Tiel's file is bigger than the GPU pool; llama.cpp ran it with the experts of 8 layers on the CPU, as in part 1.

Prompt reading went Strata's way on the two models that fit: 270 against 189 on Nail, and 279 against 191 on plain Qwen3.6 once warm, about 1.4 to 1.5 times as fast. Cold, Qwen3.6 read at 192, a tie. Generation went the other way on every model: Strata wrote at 52 to 78% of llama.cpp's speed, and on Nail, the model the agent runs below used, at 54%.

Generation also warmed up as experts were swapped in. On short prompts, Nail's generation climbed across runs from 3.4 to 9.8 to 14.3 tokens a second.

Tiel was the one I had down as Strata's best case. Its file doesn't fit the GPU, llama.cpp has to compute part of its experts on the CPU, and an engine whose whole idea is an expert cache in GPU memory should have walked that one. It lost both ways: half of llama.cpp's prompt speed, and at best 73% of its generation. I had it down wrong.

Groot didn't get that far. Strata refused to load it with "ffn_gate_exps and ffn_up_exps have different types, which this engine does not run".

The pattern over the four: faster prompt reading only when all the experts fit the GPU, slower generation always.

Three agent runs on Nail: same grades, longer walls#

Benchmarks are one thing; an agent is a long chain of requests. I ran Nail through three of part 2's tasks on Strata, with everything else as in part 2's graded Nail runs: the same GGUF file, the same prompts, the same original (v1) closing rules and the same 8,192-token reply limit. The tasks are a bookmarks REST API in Node (04), an access-log report in Java (06) and a status page in Go (07), each a fresh repository with two hours of wall time.

task	llama.cpp wall	Strata wall	llama.cpp grade	Strata grade
04 Node bookmarks API	68 min	2 h limit	2	2
06 Java access-log report	56 min	63 min, died on the context limit	3	3
07 Go status page	27 min	42 min	3	3

The grades come from the same blind flow as part 2: a script that builds and tests each project, then one small reviewer model per packet, filed under a random id, that runs its own probes in a sandbox. Grades are 1 to 5, and 3 means "needs changes".

The Java run ended the way few runs get to. Right after the agent wrote its own "BUILD SUCCESS. All tests pass", the run died on this:

request exceeds the available context size: 69652 prompt tokens, context 65536

It went out on a victory lap it had written itself, and the work was never committed.

The defects were the same kinds part 2 found, run by run:

04, Node API: crashes on a PATCH with an unknown field and on a null element in an import; its tests never exit; no README and no commit.
06, Java report: not committed, and the README's own --html report.html example crashes.
07, Go status page: an empty /metrics, and an 18.6 MB binary committed to the repository.

The grades didn't separate the two engines, which is not the same as showing equal code. Part 2 found the reviewer's scores bunch at 3, each task ran once, and the defect lists above differ from the llama.cpp runs'. What the runs do show is the time: every one took longer on Strata.

Where the agents' time went: generation#

The servers' own logs show what each run spent. Prompt speed here is taken over the requests with at least 1,000 new prompt tokens:

task	engine	requests	new prompt tokens	prompt t/s	generated tokens	gen t/s
04	llama.cpp	80	169,883	101	44,004	19.8
04	Strata	153	196,006	156	61,798	11.8
06	llama.cpp	63	159,185	91	31,965	21.8
06	Strata	31	206,265	127	26,386	12.5
07	llama.cpp	46	30,026	111	24,359	21.0
07	Strata	48	41,757	145	24,823	11.9

Across the three tasks, Strata read prompts at 1.3 to 1.5 times llama.cpp's speed and generated at 0.57 to 0.60 times. In Strata's Node run, 88 of the 120 minutes went to generation and 25 to reading prompts. An agent spends most of its time writing, so a 1.4 times faster reader can't pay for a writer at 0.57.

The 3090 bench taught me this once already, with a different engine: 2.2x faster prefill bought a 1.04x faster agent there. On the laptop, the engine with the faster reader made all three agent runs slower outright. I evidently needed the lesson twice, and the second copy came with a longer wall.

What these numbers are not#

Not averages. Each Strata measurement and each agent run is n=1, and only one model, Nail, went through the agent tasks.
Not a supported setup. HIP ran on an architecture the engine doesn't list, with ROCm from wheels the engine's docs call unvalidated for RDNA2.
Not the other machine's number. The 1.3 to 2x from the Strix Halo-class test is a different setup, and nothing here reproduces or refutes it.
Not the same drafting. Strata ran its own speculative drafting and reported how many drafted tokens it guessed right (98 of 115, in one example); llama.cpp ran Nail with MTP drafting, 2 draft tokens. Same weights, different drafting.
Not part 1's table. The llama.cpp column above is a separate llama-bench run (two repetitions and 128 generated tokens, against part 1's one repetition and 64), and it differs by up to 6% from part 1's table.
Not tried: warming up Strata's expert-usage profile across a long session, longer-context benches, and other engines' options.

llama.cpp keeps the laptop#

For coding agents on this laptop, llama.cpp on Vulkan stays the engine. Strata could pay off for prompt-heavy work with short replies on a model that fits the GPU entirely, which describes plenty of workloads and none of the three runs here. It did work on my machine, as promised. It just didn't work faster.

Filed under: local-llm, llm, ai-agents, benchmarks, llama.cpp, rocm

Sources#

My own build log, correctness check, benchmark runs, agent runs, server logs and blind grades on the laptop (local, not published), 31 October to 1 November 2026.
UnbiasedStrata README at commit 22abfdb: the lineage, the quoted bullets and closing line, the CUDA target and the correctness method.
UnbiasedStrata cmake/hip_backend.cmake at commit 22abfdb: the validated and unvalidated HIP targets.
PolyStrata and Strata: the upstream engines.