2.2x faster prefill, 1.04x faster agent

Assumes you've pointed a coding agent at a local inference server and know what prefill, decode and a prefix cache are.

Before the engine was even downloaded, the research note wrote down what would get it deleted: park it if the probe "shows < 90 tok/s at ~25k on non-copy text and < 1.2x llama.cpp on edit turns". Both legs were meant for the probe. The second, edit turns, ended up being measured with a coding agent.

The microbenchmark came back with prefill 2.2x llama.cpp at 25k. The coding agent came back 1.04x faster. Same model, same RTX 3090, same 280 W power cap, the same night. Both numbers are real. Only one of them describes the work.

The microbenchmark: 2.1-2.2x prefill at every depth#

The engine is ninfer-3090 v0.16.0, a native CUDA engine (not llama.cpp, not GGUF) with its own .ninfer weight format. I ran a third-party W4A16 build of Swift-1.5-Qwen3.8-27B in podman, with 4-bit KV cache and DFlash2 speculative decoding at 7 draft tokens. The baseline is llama.cpp with an IQ4_XS GGUF of the same model and MTP speculation at 3 draft tokens.

The probe sends the same seeded needle-in-a-haystack prompt to every engine at four depths and records prefill and decode rates. One run per depth:

All rates in tokens per second. Prefill is 2.2x at 25k, 77k and 96k, and 2.1x on the short prompt. Decode is 1.4x to 1.6x. At 25k, time to first token drops from 22.46 s to 10.15 s. At 96k it drops from 120.63 s to 55.03 s, 65.6 s saved per cold prompt. The needle was found at every depth on both engines.

If you run agents, you have read the next paragraph in a dozen release notes. Agent prompts are long, they grow every turn, and agents are prefill-bound. Halve the prefill, halve the waiting.

The park rule: decode failed, prefill bought a bench slot#

Decode at 25k came in at 74.2 against the rule's 90. That leg failed. The edit-turn leg needed an agent, so the verdict paragraph argued the engine onto the bench: fastest prefill measured, decode tied with the fastest other engine, and agent loops are prefill-bound. Two cells went into the overnight queue: three short scripting tasks (t10-t12) three times each, and a Zig ledger task at 128k context three times.

The agent bench: 1.04x and 1.03x#

OpenCode 1.18.4, Swift-1.5, medium reasoning effort, the model card's sampler, three runs per cell. Sum of wall-clock seconds:

814 over 779 is 1.04x. 2,130 over 2,066 is 1.03x. The bar was 1.2x. And open-jet, an engine whose prefill at 25k was 1,153 tok/s, close to llama.cpp's 1,098, beat both by a wide margin.

The commit after the bench read parked on speed (1.04x llama.cpp+MTP). The engine directory, the 20 GB weights file and the container image were deleted the same morning to free disk.

The hypothesis in the notes: prefix reuse#

The writeup that parked it offered an explanation and flagged it in the same breath: the 2.2x does not show up in agent walls "likely because both engines reuse the cached prefix, so most agent turns prefill only the new tokens; not measured".

That was a hypothesis when it was written. The servers logged enough to check it, after the fact.

What the server logs say: 2 to 12 percent of the prompt was prefilled#

ninfer writes one line per finished request. Here is request 18 in the server log, the fifth agent turn of the first t10 run, pasted from the bench log (timestamps are UTC):

2026-10-16 00:16:54.406  INFO  req#18 done | openai-chat | tool calls 1 | prompt 22,671 | output 50 | cache 22,650 (99.9%, checkpoint) | TTFT 193 ms | total 586 ms | prefill 132.2 tok/s | decode 124.8 tok/s | dflash2 accepted 39/82 (47.6%) | copies accepted 1/12 (8.3%)

A 22,671-token prompt. 22,650 of those tokens came out of the cache. The engine that does 2,430 tok/s on a cold 25k prompt reports a prefill rate of 132.2 tok/s here, because it prefilled 21 tokens and the overhead dominated. Across the whole bench, ninfer served 291 agent requests carrying 13,429,009 prompt tokens (304 requests counting 13 without tools, 12 of them session titles). 13,128,389 of those tokens, 97.8%, were cache hits. llama.cpp's print_timing lines record only the tokens it evaluated, so the same split falls out of its logs too. Summing every request per arm:

ninfer's prefill seconds are the sum of time-to-first-token, so they slightly overstate it. Server time is 94% to 98% of wall time in all four rows; the rest is tool execution and the client.

So the hypothesis holds as far as these logs reach. Both engines reuse the prefix, and prefill is an eighth to a sixth of where llama.cpp spends its time.

Amdahl's cap: the most 2.2x could buy#

Take llama.cpp's Zig arm, 365 s of prefill out of 2,053 s of server time, and give it ninfer's probe-measured 2.2x prefill with every other token unchanged. Prefill drops to 166 s, the total to 1,854 s: 1.11x. The short cells: 96 of 768 s, which comes to 1.07x.

Make prefill free and the caps are 1.14x and 1.22x. By prefill alone, the park rule's 1.2x bar was out of reach on the short cells for any engine, however fast. On the Zig cell it would take zero-cost prefill to clear it, by less than two hundredths.

That alone kills the "agents are prefill-bound" line, at least for a 27B model on a 3090 with a working prefix cache. But it predicts something like 1.07x to 1.11x, and the logs contain a second surprise.

The part nobody predicted: ninfer's runs said 1.7x more#

The decode column is the strange one. Inside the agent loop, ninfer decoded at 127.7 tok/s on the short cells against llama.cpp's 76.5, and at 102.7 against 65.8 on Zig. That is 1.67x and 1.56x, in the same range as the probe's 1.42x to 1.64x. Yet it spent more seconds decoding: 697 against 672, 1,781 against 1,688.

It decoded faster because it had more to decode. Output tokens:

• t10-t12: ninfer 89,044, llama.cpp 51,465, 1.73x
• Zig spec: ninfer 182,997, llama.cpp 111,093, 1.65x

This is not one runaway session. ninfer's arm generated more in 11 of the 12 sessions; the only exception was the second run of t10. Its short runs also made 85 tool calls between them, against 56. open-jet, a different engine again, generated about what llama.cpp did: 54,066 and 118,189 tokens. So the extra volume belongs to the ninfer arm, not to "any engine that isn't llama.cpp".

Why, I don't know. Among the differences from the llama.cpp arm: W4A16 group-wise weights instead of IQ4_XS, the chat template baked into the .ninfer artifact instead of the GGUF's, 4-bit KV cache instead of q8, and DFlash2 drafting instead of MTP. Both engines kept earlier reasoning in the history (ninfer via --preserve-thinking, llama.cpp by default, per its startup log), possibly through different template code. ninfer's short-cell prompts summed to 1.30 million tokens against about 0.73 million, which fits more turns, longer turns, or both. None of it was tested before the files were deleted.

The ledger for the bench, then: prefill saved 34 s on the short cells and 129 s on Zig. Decode gave back 25 s and 93 s. The rest of the wall gap is outside the server: tool execution and client time came to 20 s against 46 s on the short cells, 49 s against 77 s on Zig. The headline 1.04x is a faster engine in an arm that wrote 1.7x more.

Measure the loop, not the depth#

A fixed-depth probe asks: how long does a cold prompt of N tokens take? An agent almost never sends one. It sends a long prompt once, then dozens of prompts that differ from the previous one by a tool result. By count, the probe measured a request shape that barely occurs in the bench.

ninfer's log has one clear exception, in the first Zig run:

2026-10-16 00:42:12.528  INFO  req#175 done | openai-chat | stop token | prompt 80,193 | output 1,909 | cache 0 (0.0%) | TTFT 59.9s | total 1m 25.5s | queue 37.3 ms | prefill 1.36k tok/s | decode 74.6 tok/s | dflash2 accepted 1,266/4,517 (28.0%) | copies accepted 8/30 (26.7%)

An 80,193-token prompt, no tools, nothing cached. That is the request the probe models. Of ninfer's 304 requests, it was the only one with more than 18k uncached tokens; the next largest prefilled 17,783. It alone is a quarter of ninfer's Zig prefill seconds. llama.cpp's Zig arm had one of these too: 67,015 tokens prefilled in 74.8 s, 896 tok/s. At that rate the 80,193-token prompt takes about 89.5 s, so ninfer saved about 30 s on it, a 1.5x gap rather than the probe's 2.2x.

What the per-request logs give you, and the probe cannot:

• Prompt tokens prefilled vs cached. If the cache serves 90%+, prefill speed is a footnote.
• Prefill and decode seconds, summed. That ratio is your Amdahl cap before you bench anything.
• Output tokens per task. Tokens per second is an engine number; tokens per task is the arm's.

Both servers already wrote all three. I had never added them up before writing a park rule.

Caveats#

• One run per depth in the probe; three runs per cell in the bench. The 1.04x and 1.03x margins sit well inside per-run spread: ninfer's t10 runs took 118, 40 and 92 s.
• The llama.cpp and open-jet short cells ran at 64k context; ninfer at 128k. The largest prompt in any short cell was under 24k tokens, so neither limit was reached. The Zig cells all ran at 128k.
• Weights differ (W4A16 vs IQ4_XS vs Q4_K_M) and ninfer ran 4-bit KV against q8 on the others. On the Zig cell ninfer was the only engine with 15/15 hidden core tests in all three runs. With n=3 and different weights, that is a lead, not a result.
• The llama.cpp prompt totals are reconstructed from each request's final token count minus its output, so they are approximate. The prefilled-token counts are logged directly.

Filed under: local-llm, llm, ai-agents, benchmarks

Read next#

• Zig needed context, not time — the same Zig ledger task, and why it runs at 128k.
• The deadlock test can't see a half-held lock — what the hidden core tests on that task do and don't catch.

Sources#

• ashalliants/ninfer-3090 — the engine fork; release v0.16.0-rtx3090 (published 2026-10-14) was used.
• hamixdd/Swift-1.5-Qwen3.8-27B-W4A16-RTX3090-NInfer-v3 — the third-party W4A16 .ninfer weights.
• Probe rows, bench result rows and server logs are from my local benchmark repository (not public): the S3 speed-probe JSONL, OpenCode result rows for each arm, ninfer's container log, and llama.cpp and open-jet server logs for the matching cells.