The engine got credit for the quant

2026-10-16 · 11 min read

Contents

Assumes you've served a GGUF through llama.cpp and know what a quant is.

Two inference engines ran the same 17-gigabyte GGUF through the same nine Zig agent runs, and each finished with 7 PASS, 1 PARTIAL, 1 FAIL. A tie is a boring result. This one was the most useful number of the week, because the evening before, one of those engines had been beating the other 7 to 3.

That earlier comparison used the same model (Swift-1.5, a Qwen3.8-27B fine-tune), the same Zig task, the same agent client, 128k of context and nine runs per engine. Summed over the nine runs:

engine	PASS / PARTIAL / FAIL	total wall, 9 runs
llama.cpp + MTP	3 / 5 / 1	10,309 s
open-jet megakernel	7 / 1 / 1	7,230 s

The new engine was faster and passed more than twice as often.

I didn't write that up as a win. Under the table went a caveat: the PASS gap was not an engine claim, for three reasons. One of the three was wrong, and I fixed it thirty seconds later. The caveat as a whole was right, and it settled nothing. One more run took the credit away from the engine. Where the credit belongs instead is less settled than the clean table makes it look.

Two variables in one row#

The task is t15-ledger-zig from my local agent benchmark: add an idempotent, atomic, thread-safe transfer to a small Zig 0.16 ledger service. A scorer runs 15 hidden core tests plus detail checks; a missed core test is a FAIL, detail misses alone are a PARTIAL. Three prompt styles (spec, test-first, and spec plus a review turn), three runs each, OpenCode 1.18.4, medium effort, the model card's sampler, an RTX 3090 capped at 280 W, q8 KV cache on both engines.

open-jet is a megakernel engine for exactly one model family. It doesn't take any GGUF you hand it: it needs its own Q4_K_M tensor mix, which I built from the BF16 weights the day before. llama.cpp's profiles serve IQ4_XS, the quant every Swift launcher here uses. The two files:

Swift-1.5-Qwen3.8-27B-IQ4_XS.gguf: 15,475,951,296 bytes
Swift-1.5-Qwen3.8-27B-Q4_K_M-openjet.gguf: 17,106,773,824 bytes, 5.01 bits per weight

So each row changed two things at once, and the rows were labelled with only one of them. The table credited the engine because the engine was the column I'd set out to test.

A caveat that decides nothing#

The caveat listed three suspects for the 7-versus-3 gap:

The weights differ: Q4_K_M against IQ4_XS, about 1.6 GB more.
The KV cache differs: q8 against llama.cpp's default f16.
n is 3 per cell.

Suspect two had an alibi. The bench server passes -ctk q8_0 -ctv q8_0, so both engines ran q8. The README commit went in at 22:10:57 and the correction at 22:11:27. The conclusion stayed as it was: speed is the robust result, quality is "no worse".

"No worse" is a careful phrase. It also assigns the gap to no one. It can't be wrong, so it can't tell me anything, and it left the actual question (is it the file, or is it noise?) parked behind a hedge.

Around 23:54 a note on the same results made the obvious move. In my words: the larger quant could explain the pass count, three runs per prompt could explain it too, the speed was the solid part, and before trusting open-jet on quality I should run llama.cpp on open-jet's own Q4_K_M file. The next queue was committed at 23:55. That's the difference between a caveat and a control: one lists suspects, the other puts one of them in a different room.

Same file, other engine#

The control ran overnight behind four other queues: llama.cpp with MTP, three draft tokens, q8 KV, 131,072 tokens of context confirmed in every serve log, the same llama.cpp build as the IQ4_XS arm, and Swift-1.5-Qwen3.8-27B-Q4_K_M-openjet.gguf as the weights. Nothing else moved. It started at 04:25 and finished at 07:55.

prompt	llama.cpp, IQ4_XS	llama.cpp, Q4_K_M	open-jet, Q4_K_M
spec	1 / 1 / 1	3 / 0 / 0	2 / 0 / 1
test-first	1 / 2 / 0	2 / 1 / 0	2 / 1 / 0
review	1 / 2 / 0	2 / 0 / 1	3 / 0 / 0
all 9	3 / 5 / 1	7 / 1 / 1	7 / 1 / 1
15/15 hidden core	8 of 9	8 of 9	8 of 9
total wall	10,309 s	12,276 s	7,230 s

Put llama.cpp on open-jet's file and llama.cpp scores exactly what open-jet scored. The per-prompt splits differ by a run here and there, which is what three runs per cell look like. The totals match exactly. Whatever produced the 7, it showed up on both engines once they shared a file: the engine had been taking credit for its luggage.

Where the gap lives: freeing a string#

The three arms pass the full hidden core equally often. The gap lives in one detail check, memory: noLeaksAcrossTransferReplayConflictAndFailure, and that check is small enough to read whole:

pub fn memoryScenario() !void {
    var e: Env = undefined;
    e.initWith(std.testing.allocator);
    defer e.deinit();
    try requireShape();
    const a = try e.open(eur(10000));
    const b = try e.open(eur(0));
    var kb: [36]u8 = undefined;
    const k = keyFor(&kb, "mem-1");
    _ = try mustOk(call(&e.svc, a, b, eur(100), k), "transfer");
    _ = call(&e.svc, a, b, eur(100), k); // replay (or a second transfer when keyless)
    _ = call(&e.svc, a, b, eur(1_000_000), k); // conflict or overdraft
    var kb2: [36]u8 = undefined;
    _ = call(&e.svc, a, b, eur(1_000_000), keyFor(&kb2, "mem-2")); // overdraft
    e.store.failNextSaveOf(b);
    var kb3: [36]u8 = undefined;
    _ = call(&e.svc, a, b, eur(100), keyFor(&kb3, "mem-3")); // storage failure
    e.store.failNextSaveOf(null);
    _ = call(&e.svc, a, b, eur(100), keyFor(&kb3, "mem-3")); // retry
    for (0..50) |_| {
        var kb4: [36]u8 = undefined;
        _ = call(&e.svc, b, a, eur(1), freshKey(&kb4));
    }
}

It runs on std.testing.allocator, so anything still allocated at deinit fails the check. The scenario discards every receipt it gets back, because the task says it can: TransferReceipt is "a new plain value struct", and "everything the service allocates is freed by Service.deinit."

The check failed in 6 of 9 IQ4_XS runs and in 2 of 9 on each Q4_K_M arm (one of open-jet's two was a crash, not a leak). Every run that actually leaked reported exactly 53 leaks, and every one came from the same idea, written in four differently named functions:

error(DebugAllocator): memory address 0x7f29b3440040 leaked: 
[… std/mem/Allocator.zig frame …]
[…]/t15-score/acceptance/src/service.zig:230:38: 0x123176b in receipt (root.zig)
        const key = try self.gpa.dupe(u8, idempotency_key);
                                     ^

The model dupes the idempotency key onto the heap, puts the copy in the receipt, and hands the receipt to a caller that was promised a plain value. The function is receipt in this run, and receiptFor, receiptCopy or plain transfer in the others. The scenario hands out 53 receipts: the first transfer, the replay, the retry, and 50 reverse transfers. 53 receipts, 53 orphaned strings.

In this batch, the IQ4_XS runs failed the leak check three times as often. Whether that's the file talking is a question for the last section.

The speed stayed#

Changing the file moved the arm up and to the right. Changing the engine moved it straight left. The original table compared the two corners and drew one arrow between them.

On the same Q4_K_M file, open-jet took 7,230 seconds against llama.cpp's 12,276: 1.70 times faster. Tool calls per run overlap (63 to 150 on llama.cpp, 67 to 139 on open-jet), so open-jet didn't get there by taking a shorter path. On the short scripting tasks from the earlier probe, all 27 runs passed and open-jet still won: 629 seconds total against 814 for llama.cpp with MTP. That probe was not a same-file run either (open-jet on its Q4_K_M with four draft tokens, llama.cpp on IQ4_XS with three), but the file difference works against open-jet, since the bigger file is the slower one on llama.cpp.

The quant has a speed cost of its own on llama.cpp. Across the nine runs' serve logs, the decode progress samples average about 49 tokens per second on Q4_K_M and 59 on IQ4_XS, and the Q4_K_M arm's total wall is 19% longer (12,276 against 10,309 s). On llama.cpp, the four extra PASSes cost about a fifth more time. open-jet pays for the bigger file and still finishes first.

The same IQ4_XS file has scored 6 of 9 here#

The control matched every knob of the IQ4_XS arm, so 7 of 9 against 3 of 9 is a same-condition comparison. It is still three runs per prompt, and the IQ4_XS file has a history on these exact cells.

A few days earlier (serve logs dated 11 to 13 October), the same Swift IQ4_XS file ran the same three prompts at 128k with a four-hour limit, under two older conditions: llama.cpp without MTP, and OpenCode still loading my global ~/.claude instructions into every prompt. That arm went 6 / 3 / 0, all nine runs passing every hidden core test. The leak check failed in 2 of its 9 runs, the same rate as both Q4_K_M arms.

So IQ4_XS has scored 6 of 9 and 3 of 9 on these cells, and leaked 2 times and 6 times. MTP shouldn't change which string a model frees: speculative decoding keeps the output distribution of the model it speeds up. The global instructions might. With them, every request carried my whole global CLAUDE.md and skill list, tens of kilobytes of my own working rules, and I can't rule out that they changed how the model wrote Zig. So the earlier arm is not a clean repeat, only a warning that IQ4_XS has done much better on these cells. The statistics don't settle it either: 7 of 9 against 3 of 9 is a two-sided Fisher exact p of about 0.15, and the IQ4_XS file against itself, 6 of 9 against 3 of 9, is about 0.35. On a Spring Boot task at 128k, the two files went 1 / 2 / 0 (Q4_K_M) and 0 / 3 / 0 (IQ4_XS), all six runs passing every hidden core test: no real difference there either.

What survives all of this:

The engine does not own the PASS gap. Same file, other engine, same total.
The engine does own the speed. 1.70 times faster on the same file, 1.29 times on the short tasks.
Hidden core 15/15 in 8 of 9 runs, in all three same-condition arms.

The quant gap is a lead. The notes say so: llama.cpp stays the default engine, since it runs every model and every profile while open-jet runs one model family from one hand-built file through a sampler proxy, and the launch profiles still serve IQ4_XS. Switching the Swift launchers to a bigger, slower file on a p of 0.15 would be the same mistake with a different variable.

The caveat at 22:10 named the right suspect and decided nothing. The control cleared the engine. The quant is still only a suspect.

Filed under: local-llm, llm, ai-agents, benchmarks, zig

Sources#

All local to the benchmark repository (no public remote):

docs/openjet-20261015/README.md: open-jet probe, short-task and Zig 128k tables, quant control section.
launch/WHAT-TO-USE.md, section "Choose an engine": default-engine decision; the line that launch profiles still serve IQ4_XS.
agent-playground/bench/results-*{zmtp,zq4,zmk}*.jsonl: the 27 run rows (verdicts, walls, tool calls, hidden-core counts, failing check names).
agent-playground/bench/results-*{speclong1,tddlong1,steerrevlong1}*.jsonl and their serve logs: the earlier IQ4_XS arm (no MTP, global prompt loaded).
agent-playground/bench/serve-logs/*-zmtp*.log, *-zq4*.log: llama.cpp serve logs for both arms (model path, context, decode samples).
agent-playground/bench/scorers/hidden/t15/zz_hidden_helpers.zig: memoryScenario.
agent-playground/bench/tasks/t15-ledger-zig.txt: the task contract quoted above.
docs/openjet-swift-gguf-20261014.md: how the open-jet Q4_K_M file was built, its size and bits per weight.