Most models passed the scripts. The ledgers split them

Assumes you've driven a coding agent through a client like OpenCode. Part 1 introduced the bench; part 22 covered how fast the same models run.

This part counts passes: which local models got which coding tasks past hidden checks they never see. The models run on one RTX 3090 (24 GB of VRAM), mostly served by llama.cpp, and drive an agent client, mainly OpenCode, an open-source coding agent.

TL;DR — Since 15 October, through OpenCode on one RTX 3090, 18 models ran six small command-line tools in their first intake arm (a model's first batch of intake runs under one set of settings): 288 of 309 runs passed every hidden check, and 11 models passed all of theirs. 20 models ran feature work in existing Java, Go and Spring Boot code, again counting each model's first arm: 80 of 180 runs passed, no model passed all 9, five passed none. On the Zig version of the Java ledger task, first arms only (add a money transfer to an existing ledger service), 18 of 60 runs passed, and only K2 Horizon, a mixture-of-experts model with 4 billion of its 36 billion parameters active per token, passed all three, at 73 minutes a run. The best on feature work were tuned variants of the dense Qwen3.8-27B model with MTP drafting, the model's own draft layer (ThinkingCap 8 of 9, Qwen3.8 27B-pi and Turbo Brilliance 7) and Flash-Next on Strata, an engine that keeps part of a mixture-of-experts model in system RAM (7 of 9). A gap of one or two passes out of nine is inside run-to-run noise.

Three periods, one comparable#

My bench has run 515 batches of agent runs on 57 bench keys, the names the runs are filed under (a few keys are one model with reasoning switched off), and only the last third compares with itself.

• Before 6 October, every answer was cut at 8,192 output tokens, so reasoning models often stopped mid-thought, and most tasks ran once.
• 7 to 14 October, the output budget went up to a quarter of the context. OpenCode still carried my personal instructions file into every prompt, about 14,000 extra tokens, and MTP drafting (the model's own extra layer proposing tokens for the full model to check) was off.
• From 15 October, each OpenCode run is isolated (no personal instructions, its own database and git config), and MTP drafting is on wherever the model has it. 175 of the 515 batches are from this period.

The rankings below use only the third period. The first two get a table at the end, labelled as what they are: coverage, not comparison.

A model that joins the bench goes through an intake: a fixed set of tasks in three tiers, three runs per task (two on the CLI-tool tier since part 21, which calls that tier "basic"). A run passes when every hidden check passes. The model never sees those checks. A run that fails only a detail check is PARTIAL, one that fails a core check is FAIL, and both count here as not passed. Since 22 October, documentation checks (a README section, a diagram) are recorded but don't count (part 4).

The tasks, in plain words#

Six small command-line tools. Each is a one- or two-page spec in a different language, with traps planted in the test data.

• Compare two .env files: a Go tool that lists keys only in the first file, keys only in the second, and changed values, with JSON output, exit codes 0/1/2 and table-driven tests (t7).
• Find the slowest tests: a Java tool, JDK only, that reads Maven Surefire XML reports and lists the slowest test cases with a summary; no Maven or JUnit allowed, so it tests itself from a hand-written main() (t8).
• Report stale git branches: a Go tool that lists local branches with last-commit date, age in days, whether they're merged into main and whether they're stale, oldest first (t9).
• Summarise a messy sales CSV: a Python tool that counts rows, skips malformed records, finds duplicates and totals revenue by region (t10).
• Sort an inbox by content: a bash script that moves every file into log, csv, json or other by what the file contains, not by its name, and never loses a file (t11).
• Merge two event streams: a Node.js script that merges two JSONL streams by id, newest timestamp wins, and counts conflicts and skipped lines (t12).

Feature work in an existing codebase. The model works inside a project with its own conventions, and hidden acceptance tests grade the result.

• Add a money transfer to a Java ledger: transfer(from, to, amount, idempotencyKey) in an existing Maven module. All or nothing when storage fails halfway, safe to retry with the same key, no deadlock when two transfers run in opposite directions, with JUnit tests including a concurrent one (t13).
• The same transfer in Go: the Go port, with sentinel errors, overflow rejected and go test -race clean (t14).
• Fix a bug and add an endpoint in Spring Boot: find the real cause of a failing test (cancelled events come back, because of a read-only transaction), then add registration endpoints with validation: 201 with a Location header, 400 with a list of field errors, 404 and 409 where they belong (t16).

Zig. The same ledger transfer in Zig 0.16, a toolchain newer than most models' training data, with no memory leaks allowed under the test allocator and a four-hour limit (t15).

Two more tiers appear only in the history table at the end. The basic tier is six early chores, mostly Python: a Markdown table-of-contents tool, a CSV stats tool built over three turns, one small change in each of four real Java repositories, file chores that must not lose data, a seven-turn inventory tool, and finding the one broken function among 120 near-identical ones in a 30k-token file. The operator tier is open-ended analyst work with hidden answer keys, mostly one run each: anomalies in 50,000 payments, log triage, a PDF book turned into a learning path, a security review of 30 servlets, research from conflicting sources, a disk-usage report in pure shell, and reconciling a flaky sales API.

The ranking since 15 October#

One row per model, its first intake arm on each tier (an arm is one batch of runs under one set of settings), OpenCode client. Passed runs over runs:

A dash means the model ran no intake arm on that tier since 15 October. Apart from Flash-Next, the top eight are the dense Qwen3.8-27B and its tunes and quants. Flash-Next is Qwen3.8-Flash-Next at 2 bits, a mixture-of-experts model run on Strata, an engine that keeps a cache of experts on the GPU and the rest in system RAM (part 1). The GSQ files are Qwen3.8-27B and Swift-1.5 in a published 3-bit quantization (GSQ-RCO IQ3_S, 12.1 GB) instead of the usual IQ4_XS. Rows marked * ran some or all of their arms at 64k context. "35B-A3B" means 35 billion parameters in total, about 3 billion active per token.

Read it with its limits:

• Not like-for-like. Each model ran its own card's sampler, except Flash-Next IQ2_XS (its engine's default, not recorded). Context was 128k except for the * rows; Flash-Next Q2_0's isn't in my summary. Engines differ:

  • two builds of llama.cpp, the newer one from 26 October, for most models;
  • separate llama.cpp builds for Bonsai (Prism's fork), Flint and K2 Horizon;
  • a newer engine build my summary doesn't name for Flash-Next IQ2_XS;
  • the Strata engine for Flash-Next Q2_0.

  Reasoning effort was "medium" for the Qwen3.8 tunes and Bonsai, off for Ornith, "high" for K2 Horizon.

• Noise. In part 21, three identical runs disagreed on pass or not-pass in 34% of hard-tier groups and 56% of Zig groups. 8 of 9 against 7 of 9 is not a ranking you can lean on. 7 of 9 against 0 of 9 is.

• Flash-Next Q2_0 ran its intake under arm names my summary tables don't match, so its row comes from the per-period totals, which hold exactly one OpenCode arm per tier for it, and it has no per-task breakdown.

Some models ran a second arm on the same tier, and the hard tier mostly repeated itself. Swift-1.5 on the newer build, at the default reasoning effort instead of "medium": 12/12 on the tools, 6/9 on feature work, the same 6 as before. The two GSQ files with two MTP drafts and a cooler sampler: 12/12 and 6/9 each, the same count, though not the same passes task by task, plus single Zig runs (Qwen3.8 GSQ passed, Swift GSQ didn't). Bolt at 128k context instead of 64k: 5/9 against 4/9, the rerun part 21 picked apart.

Task by task: where the models split#

The chart counts models, not runs: for each task, how many passed every run, some, or none, summed over each model's intake arms. Flash-Next Q2_0 isn't in it (no per-task breakdown). Neither is Nanbeige on the tools: its nine runs don't split evenly over six tasks, and only one of them, an event-stream merge, passed. At least three of the others hit the time limit.

The CLI tools: nearly everyone#

Merging two event streams was passed in every run by all 16 models. Comparing .env files, summarising the sales CSV and sorting the inbox each had one model that passed only some runs. The misses are specific: Ornith counted 10 rows of the sales CSV where 8 were valid, and didn't skip the two malformed ones. K2 Horizon's .env tool left keys out of its default output and once exited 0 with differences present.

The slowest-tests report was the trap of the tier: 12 models passed every run, 3 passed some, and Bonsai passed none. Bolt's one miss and at least two of Bonsai's three are the same failure: the --json flag printed nothing a parser could read, and those two Bonsai runs also hit the time limit. Laguna-XS's miss was a rounding: 12.8 seconds where the check wanted 12.75.

Feature work: the Spring task nobody finished every time#

The Java ledger transfer split the models down the middle: 9 of 19 passed every run, 6 passed none. The Go port was harder: 6 every run, 7 some, 6 none.

Five models passed neither ledger in any run and got 0 of 9 on the whole tier: Laguna-XS, Muse-Glimmer, Ornith, Nemotron Lightning and Qwen3.6, four of them among the fastest models in part 22. Laguna-XS, Muse-Glimmer, Ornith and Qwen3.6 fail the same core check in their first recorded run: when saving the destination account fails, both balances must stay as they were. Nemotron Lightning fails its twin on the source side. That's the rollback part 14 traced run by run.

The Spring Boot task is the one no model passed in every run. 13 of 19 passed none; the six that passed any are five Qwen3.8-27B tunes and quants (Qwen3.8 27B-pi and ThinkingCap 2 of 3 each, Turbo Brilliance 1 of 3, the two GSQ files 1 of 6 each) and Bonsai (1 of 3). For the Qwen3.8 family the miss is mostly one detail check: the 400 response should carry one error entry per invalid field. Of the family's 23 Spring runs that didn't pass, at least 17 are PARTIALs on exactly that check and one is a FAIL on email validation; the result notes I have are cut short for the other five. Groot's Spring misses are FAILs on core checks instead, email validation and the duplicate-email conflict.

Zig: one clean sweep, by the slowest model#

The Zig ledger had one model pass all three runs: K2 Horizon, at 4,404 seconds per run and the slowest generation of any model that ran it (part 22: 24.7 tokens per second). Nine passed none: Laguna-XS, Muse-Glimmer, Nail, Ornith, Nemotron Lightning, Qwen3.6, Bolt, Groot and the plain Qwen3.8 27B. The Qwen3.8 tunes mostly landed 1 or 2 of 3, and several of their misses are PARTIALs on the memory check: a leak somewhere across a transfer, a replay, a conflict and a failure.

Since 15 October, without a full intake#

These models ran since 15 October but not the full intake, so they stay out of the ranking:

The Q4_K_M row is open-jet's file served by llama.cpp, the control from part 7.

The operator tier ran since 15 October too, mostly through Pi, a minimal agent client, and mostly once per task: Flash-Next Q2_0 10 of 14, Qwen3.8 27B 7 of 19, Swift-1.5 7 of 22, Qwen3.8 27B-pi and ThinkingCap 5 of 22 each, Nemotron Lightning 5 of 21, Muse-Glimmer 4 of 7, Ornith 4 of 9, Laguna-XS 2 of 7. With one run per task and mixed clients, those are a sketch.

Before 15 October: not comparable#

Every model ever benched, before the rules settled. Answers were cut at 8,192 tokens before 6 October, and the 7-14 October rows carried my instructions file and ran without MTP. The bench keys are as the bench named them. Passed runs over runs, all arms and clients summed.

Before 6 October (42 bench keys):

7 to 14 October (14 bench keys):

28 of these bench keys never ran after 14 October, and a 0 here can mean a model cut off at 8,192 tokens mid-thought as easily as a model that couldn't do the task. Five small models (Ministral 3B, SmolLM3 3B, LFM2.5 2.6B, LFM2.5 8B-A1B, GLM-4 9B) passed nothing in any run, which is the one result in these tables I'd carry forward. So did Gemma 4 agentic v2, before and after 15 October (0 of 22).

What I'd run on a 24 GB card#

From the ranking above, with its noise attached:

• Feature work in a real codebase: a Qwen3.8-27B tune with MTP drafting on llama.cpp at 128k. ThinkingCap (8 of 9), Qwen3.8 27B-pi and Turbo Brilliance (7 of 9) or Swift-1.5 (6 of 9, twice). All four passed every CLI-tool run in their intake. Between them I'd pick by speed or taste; the bench can't separate them.
• Less memory to spare: the GSQ IQ3_S files, 12.1 GB. Both passed every tool run (18 of 18 in the first arm, 12 of 12 in the second) and 6 of 9 on feature work in both arms, the same as Swift-1.5's 15.5 GB file.
• Scripts and small tools, speed first: the quickest first tool arms were Laguna-XS at 92 seconds a run (16 of 18), Swift-1.5 GSQ at 97 (18 of 18), Ornith at 103 (16 of 18), Swift-1.5 at 118 and Qwen3.8 27B-pi at 124 (18 of 18 each), on two llama.cpp builds. The 27Bs pass everything and also do feature work, so they're the pick here too. Laguna-XS and Ornith got 0 of 9 on feature work, as did Muse-Glimmer (18 of 18 on the tools at 219 seconds), Nemotron Lightning and Qwen3.6: not for code that moves money.
• A fast MoE that also does feature work: Groot 19B-A3B, 16 of 18 on the tools and 6 of 9 on feature work. Among the other MoEs with 2 to 4 billion active, K2 Horizon managed 4, Flint 3, Nail 1, and Laguna-XS, Ornith, Nemotron Lightning and Qwen3.6 none.
• Zig 0.16: nothing reliable. K2 Horizon's 3 of 3 came at 73 minutes a run.
• Flash-Next on Strata came within one pass of the best 27B at 7 of 9, but it needs its own engine and, per part 1, nearly all of the box's 32 GB of RAM.

The numbers say which models finished. A later part goes through how the rest failed, run by run.

Filed under: local-llm, llm, ai-agents, benchmarks, opencode

Read next#

• On one RTX 3090, agent speed followed active parameters — part 22, the speed of the same models.
• The hard tier sorted by a rollback, not by speed — the rollback check the fast MoEs keep missing.
• The third run mattered where it cost the most — how noisy one, two and three runs are, by tier.
• The hidden test asked for more than the prompt — when a hidden check turned out stricter than the task.

Sources#

• My own bench data (local, not published): per-arm results for every model since 15 October with their settings and result notes, per-task pass counts, and per-period totals for every model ever benched, as of 30 October 2026.
• The earlier parts of this series linked above, for the facts cited from them.