The hard tier sorted by a rollback, not by speed

Assumes you know how this bench grades: a prompt, a repo, and a hidden suite the agent never sees, where one failing core test makes the run FAIL.

On 19 October this post had a title, "Fast models failed the hard tier", and three runs per model to back it. The two fastest decoders on the box had just gone 0 of 9. So had Muse Glimmer, whose label in my model notes read "slow; never failed on scripting or dev tools", although the same row already recorded a failed Java ledger in all four earlier arms, each for one missing rollback. I parked the draft until the remaining queues finished, so it would not age into "still running".

They finished on 23 October, with six more models on the hard tier. The title did not survive. A 2-bit model at about 95 tokens per second passed 7 of 9. What did survive is the thing underneath the title: one rule, stated in the prompt in plain words, that most of the failing runs got wrong in exactly the same way.

The hard tier: three ledgers and a Spring endpoint#

The hard tier is three tasks, three runs each, in OpenCode with a one-hour limit: a money transfer for a small Java ledger (t13), the same ledger in Go (t14), and a registration endpoint for a Spring Boot app (t16). A Zig version of the ledger (t15) runs separately, with four hours per run and 128k context for almost every model. Every ledger prompt states the same rule. From the Java one:

- All or nothing. Either both balances change or neither does — including when the repository
  throws part-way through (it is the seam for the JDBC adapter and can fail on any call).

A few lines later it adds that a failed attempt "leaves no trace", so the client can retry with the same key. The hidden suite checks both, and a failure in either makes the run FAIL.

Unlike the blank-key check from part 13, where the hidden test asked for more than the prompt did, this check is the prompt, word for word.

Decode speed against passes on the hard tier#

Decode speed is token-weighted over every request in each model's hard-tier server log; pass counts are from the result rows after the 22 October rescore, which moved README and formatter checks out of the verdict.

The five fast MoEs passed 1 of their 45 runs between them. Muse, the second-slowest, failed all nine. The slowest, K2 Horizon, passed four. Speed does not sort this table.

Muse scored 94 of 100 and failed#

Muse's first Java run is the cleanest example of what went wrong. Its score sheet reads Verdict: FAIL — points 94/100: it built, it kept every old test green, its own tests caught all six seeded mutants. The points are a reading aid; the verdict comes from the hidden core checks, and it missed two.

The transfer itself is careful. It takes a lock per idempotency key, then both account locks in a fixed order so opposite transfers can't deadlock, then checks currency:

AccountId firstId = fromId.compareTo(toId) <= 0 ? fromId : toId;
AccountId secondId = fromId.compareTo(toId) <= 0 ? toId : fromId;
ReentrantLock firstLock = lockFor(firstId);
ReentrantLock secondLock = lockFor(secondId);
firstLock.lock();
try {
    secondLock.lock();
    try {
        Account fromAccount = load(fromId);
        Account toAccount = load(toId);
        if (!fromAccount.currency().equals(amount.currency()) || !toAccount.currency().equals(amount.currency())) {
            throw new CurrencyMismatchException(amount.currency(), fromAccount.currency());
        }
        Account fromUpdated = fromAccount.debit(amount);
        Account toUpdated = toAccount.credit(amount);
        accounts.save(fromUpdated);
        accounts.save(toUpdated);

Then two bare saves. The hidden test makes the second one throw:

source changed although the transfer failed ==> expected: <100.00 EUR> but was: <75.00 EUR>

The source is debited, the destination never credited, and 25 euros are gone. The retry check failed next to it: after a failed attempt, retrying the same key left the source at 80.00 EUR where the test expected 90.00. Locking, lock order, key replay, currency checks: right. The one failure the prompt spelled out, missed.

One of those six mutants is the scorer's own non-atomic transfer, which leaves the source debited when a save fails. Muse's tests "caught" it with a test about reusing a key with different parameters, which happened to break when the mutant was applied. None of its tests makes the repository throw.

The same rollback miss, run by run#

Count every ledger run per model: three Java, three Go from the hard tier, three Zig. A run is "core green" when every hidden core test passed, whatever happened to the details.

Qwen3.6 missed it in all nine runs, Laguna in eight. The fast MoEs' other twelve runs split into 6 green, 4 that failed a different core check, and 2 with no working code at all, one of them Ornith's Go run that added a method to an exported interface so the hidden test store stopped compiling.

The class is not a law. Nail, a Qwen3.6 tune, came out core green in three of its nine ledger runs, the rollback included. It still passed only 1 of 9 on the hard tier: two of those green runs were Java partials over a self-transfer detail check, and none of its Spring runs passed.

Flash-Next on Strata kept the catch block#

Flash-Next is the model that passed Zig in OpenCode in part 1, a 2-bit GGUF of a mixture-of-experts model with 512 routed experts per layer, 10 active per token, served by the Strata engine with its MTP draft. It is not a peer of the 30B MoEs: Hugging Face's metadata puts the GGUF at 177B parameters, and Strata keeps an expert cache on the GPU and the rest in system RAM. Its first Java run, at the same point in the transfer:

Account sourceAfter = source.debit(amount);
Account destinationAfter = destination.credit(amount);

boolean sourceSaved = false;
try {
    accounts.save(sourceAfter);
    sourceSaved = true;
    accounts.save(destinationAfter);
} catch (RuntimeException failure) {
    if (sourceSaved) {
        try {
            accounts.save(source);
        } catch (RuntimeException rollback) {
            failure.addSuppressed(rollback);
        }
    }
    throw failure;
}

Restore the source snapshot, keep the rollback's own failure attached, rethrow. All nine of its ledger runs were core green.

On the hard tier it passed Java 3 of 3, Go 3 of 3 and Spring 1 of 3. The two Spring misses failed one detail check, one error entry per invalid field, the same check that sank most of the 27B tunes' Spring runs. That puts it at 7 of 9: below ThinkingCap's 8, level with Qwen3.8-27B-pi.

Its 462 s per run is the fastest of any model that passed more than one hard run; Swift-1.5 is next at 507. Laguna, Nemotron and Nail finished faster, at 340, 357 and 406 s, with 0, 0 and 1 passes.

One condition differs, and it is on record. Strata served Flash-Next at 64k context, while the 27B tunes ran at 128k. Five of the six failing 30B-class models also ran at 64k, so the context doesn't explain the gap to them.

On Zig it passed 2 of 3, with all 15 hidden core tests green in all three runs; the third missed one detail check on blank keys. Only K2 Horizon did better in this round, at 3 of 3. At 918 s per run, Flash-Next was the fastest Zig arm with any pass, ahead of Swift-1.5 at 1005 s; Laguna, Muse and Nail were quicker and passed none.

Slow models didn't buy a pass either#

K2 Horizon ran with the experts of 24 layers on the CPU to fit 128k context into 24 GB, and decoded at 23 tokens per second. It passed 4 of 9 hard runs at 2196 s per run; all three Spring runs fell short only on the one-entry-per-field check, the third also at the one-hour limit. Then it passed all three Zig runs, the only model to do that in this round. Weak on Spring, strongest on Zig.

Bonsai 2 passed 4 of 9 at 2473 s per run, about five times Swift-1.5's time. The runs that ran out the hour failed no core check; they ran out with details unwritten.

Flint passed 3 of 9, all on the Java task. Part of that is not code at all: in two of its twelve hard runs, mistyped workspace paths got its writes refused, so almost none of the work reached the disk, and its closing summaries blamed the tools. That has its own post.

Muse, at 37 tokens per second, failed the rollback in eight of nine ledger runs. Slow and careful-looking is not the same as careful.

The operator tier, one run per task#

The operator tasks (u1–u7: log forensics, a security audit, reports from data files) run in Pi, one run per task, so they are indicative only. Flash-Next passed 6 of 7 at 128k context with a 32k output budget, the same conditions as the 27B tunes, whose best was 5 of 7. Like every local model on the bench, it failed the log-forensics task.

An earlier run at 64k context with a 16k output budget passed 4 of 7. One task there died on a context overflow, prompt (68463 tokens) leaves no room to answer in the context (65536). Context and output budget changed together, with one run per cell on each side, so I don't attribute the 4-to-6 gap to either.

What nine runs per model can say#

• Three runs per cell, one client. Every hard-tier number here is OpenCode, one engine per model. A pass rate of 7 of 9 against 6 of 9 is not a ranking.
• Speed labels came from easier tiers. Muse's "never failed" and Laguna's "fastest" were earned on scripting and dev-tool tasks. They still hold there. They said nothing about the hard tier; the one warning was already in Muse's row, a failed Java ledger in all four earlier arms, the same missing rollback, and I read the label instead of the row.
• The line ran through one check. It did not run between fast and slow, or between the 30B-class MoEs and everything else: Nail got the rollback right in 4 of its 9 ledger runs, Muse in 1. It ran between runs that wrote the catch block and runs that didn't.

Filed under: local-llm, llm, ai-agents, benchmarks, java

Read next#

• I blamed OpenCode; a 2-bit model passed Zig in it — part 1, the first time Flash-Next on Strata beat the 27B tunes, and why it is not their peer.
• The hidden test asked for more than the prompt — the hard-tier check that was unfair, and the rescore that fixed it.
• Flint mistyped its path and blamed the tools — why two of Flint's hard runs left no code to grade.
• The model card said think harder. The audits got worse — the same hard tier at xhigh reasoning effort.

Sources#

• Qwen3.8-Flash-Next GSQ-RCO GGUF — model card: 512 routed experts per layer across 48 layers, 10 active per token; the Hub's metadata gives the parameter count.
• Laguna XS 2.1 — model card: 33B total, 3B activated per token.
• Muse Glimmer 30B GGUF — model card: about 29.6B parameters including the vision encoder.
• Benchmark artifacts (local, not published): result rows for the hard tier (t13/t14/t16, 18–22 October), Zig (t15, 18–23 October) and operator tasks (u1–u7); each run's score sheet, hidden-test log and workspace diff; the server logs the speeds come from; the task prompts; the 22 October rescore.