Five languages, two reps: the round-two ranking

Builds on Nine accusations, one lie, which covers how round two was graded. This post is what the grades add up to.

Round two of my local-model coding study is finished, and this post ranks its six setups by how their code holds up when a save fails. The study asks whether a model served locally, on my 128 GB AMD Strix Halo box, can do a developer's job as a coding agent. Round two gave the six setups (a setup is one model with its quantization and inference engine) the same seven coding tickets, each in two agent harnesses, opencode and Claude Code, the programs that turn a model's replies into file edits and shell commands. Five of the tickets are the core: the same money-transfer feature for a small ledger service, in Java, Go, Zig, TypeScript and Python.

TL;DR — 144 agent runs, with each core ticket run twice per harness. Flash-Next (Qwen3.8-Flash-Next, a mixture-of-experts model) holds the top two places in both of its quantizations (how many bits each weight gets): IQ4_XS passed 97 of 108 hidden failure-path tests (tests the model never sees, whose fake store fails saves on purpose), Q2_0 passed 93 and was the only setup to pass all 20 of its core runs. Swift (a Flash-Next fine-tune) and ISTA-Coder (a Flash-Next Coder variant with half its experts pruned, run at 131072 tokens of context, half of everyone else's) tie on failure-path tests at 86; K2-Horizon (81) and Ornith (80), the two models not built on Flash-Next, are last, with K2 killed by the time limit in 13 of its 24 runs. The ranking's order inside the middle and the bottom rests on one test or a tie; what holds under every sort order I tried is the bands: both Flash-Next quantizations in the top three, K2-Horizon and Ornith in the bottom three. Across all six, the bugs follow the ticket: the Java ticket's failure-path tests pass 64 of 96, the Python ticket's 114 of 120, and the same mistake keeps turning up in the same language: the audit write after both saves in Java, the rollback on a cancelled context in Go, the swallowed rollback in Zig. Four of the six setups share Flash-Next's base, so the evidence beyond it is K2-Horizon, the one setup from outside Flash-Next's Qwen line, with the Java bug in 3 of its 4 Java runs; Ornith, a separate base model, has it in all 4.

Round two: 144 runs, one sort key#

Round two of my local-model coding study closed its headline table at 09:14 on 2026-10-30, with 144 runs and 106.7 hours of agent wall clock behind it. Four side-table reruns kept the box busy until 12:50. Now comes the part everyone wants and I trust least: the ranking. An ordered list always looks surer than the data under it, and I chose this list's sort key myself, so every place in it where one test decides the order is on me.

The box is the A9, my 128 GB AMD Strix Halo test machine (a GEEKOM A9 Mega: Ryzen AI Max+ 395, 128 GB of soldered LPDDR5X-8000). The question the study asks: can a model served on it, locally, do a developer's job as a coding agent? Each model with its quantization and inference engine is an arm. Each arm gets the same tickets: add an all-or-nothing, idempotent money transfer to a small ledger service, once each in Java, Go, Zig, TypeScript and Python (the core), plus two wide tickets, a MIPS assembly task and a Spring app. Every ticket runs in two agent harnesses, opencode and Claude Code, the programs that turn a model's replies into file edits and shell commands. Core tickets run twice per harness with a 90-minute wall; the wide ones once, with 90 minutes for MIPS and 3 hours for Spring. One run is a cell: 20 core and 4 wide cells per arm.

Three graders look at every cell:

• A scorer runs hidden tests the model never sees, including a failure-path suite whose fake store fails saves on purpose: is a failed rollback reported, does a retry after a late failure move the money once, are waiters released. The five core tickets carry 4, 6, 7, 5 and 5 of those tests, 108 per arm.
• A blind reviewer (Claude Haiku) reads the ticket, diff and code without the model's or harness's name and scores 1 to 5 as a merge decision: 2 is major rework, 3 needs changes, 4 merges after nits. Cells under 40 points are not reviewed.
• An honesty checker labels the agent's claims; part 26 is about how often both judges were wrong.

I sort by failure-path tests first, then the review mean, then the core pass rate. That order is mine. The ticket text, in all five languages, says why the feature exists at all: mobile clients that retry on a timeout have moved the same money twice. The failure-path tests are the ones that go looking for that.

The six arms, sorted#

Strata is an inference engine of its own, separate from llama.cpp. Swift-1.5 is a Flash-Next fine-tune and ISTA-Coder a Coder variant of Flash-Next with half its experts pruned, so four of the six arms share one base model. Ornith is a different base model, though its makers built its line on Qwen3.5 and Gemma 4, which makes it a cousin of Flash-Next's Qwen line. K2-Horizon is the one setup from outside that line. All six sampled with the settings their model card or GGUF metadata asks for: temperature 1.0, top-p 0.95, top-k 20, except K2 at top-k 0. Five were served at 262144 tokens of context. ISTA-Coder ran at 131072, because its engine configuration has no 256k variant, and Claude Code compacted its conversation at about 98-101k tokens in 11 of ISTA's 12 Claude Code cells. Every ISTA number below carries that.

The generated report behind this table counts 9 core passes for K2 and 13 for Ornith. It also counts cells marked "complete at the wall": killed by the time limit, verdict FAIL. I quote only real PASS verdicts.

Before reading the table as a verdict on the models, note that the arms differ in model, quantization and engine at once. The table ranks setups. Flash-Next IQ4_XS and Q2_0 are the same model on two quantizations and two engines, so that pair changes two things at once.

Where the sort key decides, and where it doesn't#

Two places in this table are decided by almost nothing.

Swift and ISTA-Coder tie: 86 of 108 failure-path tests each. The review mean breaks it, 2.74 to 2.47, and ISTA ran with half the context of everyone else.

K2-Horizon sits above Ornith by one test, 81 to 80, with 13 of its 24 cells killed by the wall and the lowest review mean of the round. I'd like a word with that row. Sorted by review instead, the order becomes IQ4_XS, Swift, Q2_0, Ornith, ISTA, K2: Swift and Q2_0 swap, Ornith climbs from sixth to fourth, K2 drops to last. Sorted by core PASS, Q2_0 leads with 20 and IQ4_XS is second.

What doesn't move under any of the three keys: Flash-Next IQ4_XS and Q2_0 are in the top three, and K2 and Ornith are in the bottom three. That is the ranking I'd defend. The places inside each band are a coin with a sort key painted on it.

It agrees with the stage-one ranking, which found the same pair: IQ4_XS reviewed better, Q2_0 was faster. Round two points the same way on review (2.85 against 2.70) and on failure-path tests (97 against 93), and Q2_0's 24 cells took 8.2 hours of agent wall clock against 18.3 for IQ4_XS's. Same caveat as then: quantization and engine changed together. Part 28 pulls engine and weights file apart for speed, as far as round one allows.

Same tickets, same bugs#

Read the table per ticket instead of per arm, and the tickets explain more of the spread than the models do. Over all six arms:

The per-ticket figures are summed from the generated report's cell list (a cell that never built counts 0 of its ticket's tests: ISTA's 8-point Go cell is 0 of 6); they add up to the arm totals above (523 of 648 failure-path tests, 83 of 120 core passes, 117 core reviews). One review in 117 core cells is a 4. The defects below come from the reviewers' blockers. Some arms have a hand count per defect, some only name the class; I give the count where one exists and say so where it doesn't.

Java: the audit write after both saves#

23 of the 24 Java cells scored 97 points or more. I read 100 points as a dare, and 97 doesn't calm me down either. Their failure-path tests passed 64 of 96, the worst ticket of the five, and three cells passed 1 of 4 (Q2_0 opencode rep 1, Ornith opencode rep 2, Ornith Claude Code rep 1). Ornith's four Java cells scored 98 or 99 and none of them is a PASS.

The defect is one line placed too late. The service saves both accounts, then writes the audit record, and nothing undoes the two saves when the audit write fails. The client sees an error, retries with the same idempotency key, and the money moves twice. Counted: 3 of 4 IQ4_XS Java cells, all 4 ISTA Java cells. Those two share Flash-Next's base, so the cells that matter more come from elsewhere. K2-Horizon, the one setup from outside the Qwen line, has it in 3 of its 4 Java cells; its fourth appends the audit events before the saves instead. Ornith, a separate base model, has it in all 4 of its Java cells, and in 3 Zig cells as well. Swift's and Q2_0's reviews name the defect without a per-language count.

Go: rolling back on a context that was already cancelled#

Go had the best failure-path rate of the three tickets below 90 %, and the defect most specific to Go is this one. When the caller cancels the request halfway through a transfer, the rollback runs on the same cancelled context, so the restore fails on the same cancellation. Counted: 4 Ornith cells, 4 K2 cells (all of K2's cancelled-context cases are Go), 3 ISTA cells. Part 15 found this one in round one.

Another Go defect is a cleanup line in one Q2_0 cell that deleted a per-key mutex while a caller was still waiting on it; part 25 has the whole story. Ornith wrote the same cleanup in Python, in one cell: a per-key lock entry deleted while held.

Two Go cells failed for reasons that aren't Go. A Swift opencode cell ended at 29 points after a malformed tool call, written out as text, stopped the session. An ISTA Claude Code cell ended at 8 points when the harness's automatic compaction came back empty: on a 52,492-token prompt the model returned 35 tokens of thinking and no answer. That death has more to it than fits here.

Zig: catch {}#

Zig passed 12 of 24 cells, the fewest, and its review mean, 2.30, is the lowest of the five core tickets. The bug reads like a decision. A failed rollback gets swallowed, in ISTA's cells and Ornith's reruns by a literal catch {}: the undo fails, the error disappears, and nothing tells the caller that the books are now wrong. Counted: every IQ4_XS Zig cell, all 4 ISTA Zig cells, and both Zig reruns in Ornith's side table. Q2_0's Zig cells have it as a named class. If consistency were the metric, ISTA's Zig would top the table: four cells, four empty catch blocks.

K2 and Ornith passed no Zig cell between them. Five of those eight cells were killed by the wall.

TypeScript and Python: the suites that pass almost everything#

TypeScript and Python passed 113 and 114 of their 120 failure-path tests, and still only one of their 48 reviews is above a 3. Ornith left a stray audit event behind in 3 TypeScript cells. In Python, the undo stops at the first failed restore: all 4 ISTA Python cells, and Ornith's in a blocker and a major. The suite passed ISTA's Python cells 18 of 20.

So of my graders, one is soft on Python, and it's the one I wrote: the failure-path suite. I'd file a complaint, but I'd be filing it with myself.

The wide tickets: MIPS and Spring#

Each arm ran each wide ticket once per harness: 12 MIPS cells, 12 Spring cells. MIPS passed 8, Spring 4; both have a review mean of 3.40 over their 10 reviewed cells. Twelve of round two's 13 reviews of 4 are here; the thirteenth is a TypeScript cell.

The low ones carry the stories. K2's Claude Code MIPS cell got a 1 for four assembly bugs and a CRC that was wrong for every non-empty buffer. K2's opencode Spring cell got a 1 because the Idempotency-Key header never binds, so every POST returns 400, and transfer() calls its own @Transactional method, so no transaction runs; part 22 has K2's first round with that second trap. Ornith's Claude Code Spring cell turned a reused key on a second account pair into an unhandled DataIntegrityViolationException in 20 of 20 runs, instead of a 409. The money moved once; the error is what the client got. ISTA's opencode Spring cell ended at 20 points with its reasoning degenerated into " Spring" repeated 256 times, about 34k tokens in.

The bottom of the table is partly a wall clock#

K2-Horizon was killed at the wall in 9 of its 20 core cells and all 4 wide ones. Ornith lost 4 cells to it, all in Claude Code, out of 12 Claude Code cells; no other arm lost any. My numbered rules were about test rigs until now; rule nine is about clocks: a run killed at the wall records when the time ran out and grades nothing. So K2's 2.21 averages over unfinished work: 8 of the 19 reviews behind it are of cells stopped mid-work. Ornith's Claude Code mean is 2.40 with its 3 killed core cells and 2.57 over the 7 without them.

Ornith's four killed cells were rerun once each with a 4-hour wall, as a side table that stays out of its headline:

None of the four reruns reached the old wall: they finished in 41 to 63 minutes, inside the 90 they had been killed at. So these are new samples of the same cells, not proof that 90 minutes was too short, and the scores moved both ways. The TypeScript cell went from 99 points to 71. All four reruns are overclaims, none a lie cell.

How much one rerun moves#

Each core ticket runs twice per harness: 10 pairs of cells per arm. Counted from the generated report, where "flip" means the verdict changed between rep 1 and rep 2, and a narrower flip means it crossed PASS / not PASS:

Half of ISTA's and K2's pairs disagree with themselves. Q2_0 didn't flip once, which is also the arm that passed every core cell; there was not much room to flip into.

Honesty, in one paragraph#

Over the 137 reviewed cells of the six arms, the filed honesty labels (the checker's labels, after the corrections a raw-session check forced) come to 39 honest, 97 overclaims (attempted, but wrong) and 1 lie cell, K2's Go cell whose summary claims eight tests including a statements test, where seven exist and none tests statements. No model is flagged LIAR, which takes two lie cells across all of a model's arms; K2 has 1 in 31. Overclaim shares run from 10 of 24 (IQ4_XS) to 20 of 23 (Ornith), and an overclaim counts as a bug and never as a lie, by the rule. Part 26 re-reads all 14 "false" verdicts of the round.

What this table leaves out#

• Harness comparisons are soft. A blind re-review of 12 cells earlier in the study moved Claude Code's mean by -0.33 and opencode's by 0: the packets had shown which harness ran. Every opencode against Claude Code gap in round two is smaller than that (Ornith's is the largest: 2.60 against 2.40). Both harnesses also loaded my global CLAUDE.md instruction file; only Claude Code loaded my rules files.
• The reviewers added a probe of their own. A "write lands, then raises" probe that isn't in the review brief was the merge blocker in 0 of 23 Swift cells, 3 of 24 IQ4_XS, 2 of 24 Q2_0, 0 of 23 Ornith, 0 of 21 K2, 1 of 22 ISTA, and 0 of 4 side-table reruns. Without those cells IQ4_XS's mean is 2.88 over 17, Q2_0's 2.72 over 18. Part 26 has the details.
• One review per cell, on a whole-point scale. A mean of 2.70 is fourteen 3s and six 2s; the second decimal comes from the division.
• ISTA-Coder ran at half the context, with Claude Code compacting in 11 of 12 cells.

Five languages and six setups, and the finding I'd bet on sits in no single row. On these tickets the bug follows the ticket, across every base model the round had. The table will regenerate next round looking exactly as sure of itself as this one, and I already know which Java line I'll read first: the one after the second save.

Filed under: ai-agents, local-llm, benchmarks, testing

Read next#

• Two rounds, seventeen arms: the engine set the speed, not the quant: how fast every arm of both rounds ran, and how much of the Q2_0 / IQ4_XS gap the engine explains.
• Nine accusations, one lie: re-reading round two's judges: how often the reviewer and the checker were wrong, in both directions.
• The best-reviewed code came from the slowest Flash-Next arm: the stage-one ranking and the same IQ4_XS / Q2_0 pair.
• The lock map deleted a mutex someone was waiting on: the Go defect that passed 6 of 6 failure-path tests.

Sources#

• Benchmark artifacts (local, not published): round-two matrix rows, hidden-test and failure-path results, blind review and honesty-check outputs for the six arms and the side table, per-arm defect hand counts, 2026-10-25 to 2026-10-30.
• The study's generated report (cell list, per-arm and per-model tables), built 2026-10-30 13:19 EET.
• unsloth/Qwen3.8-Flash-Next-GGUF -- model files for the Flash-Next IQ4_XS arm.
• ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF -- model files for the Flash-Next Q2_0 arm.
• ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF -- model files for the Swift-1.5 IQ3_S arm.
• ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF -- model files for the ISTA-Coder IQ1_M arm.
• IFM/K2-Horizon-MoVA-36B-A4B-GGUF -- model files for the K2-Horizon-MoVA-36B-A4B Q8_0 arm.
• ornith-ai/Ornith-1.5-35B-A3B-GGUF -- model files for the Ornith-1.5-35B Q8 arm.