The third run mattered where it cost the most

· 8 min read
Contents

Assumes you've driven a coding agent through a client like OpenCode. Part 1 introduced the bench.

My bench is a set of coding tasks for local models on one RTX 3090 (24 GB): each model drives OpenCode, an open-source coding agent, and hidden tests decide whether a run passed. Until this part, a new model went through an intake: every task three times, in three tiers. The basic tier is six small scripting tasks, the hard tier three ledger services in Java, Go and Spring with hidden core tests, and the Zig tier one Zig ledger task that may run for four hours. One model's full intake cost about a day and a half of the card's time, and most of that is Zig.

TL;DR — Over 739 groups of three identical agent runs (one model, one task, three runs), the runs disagreed on pass or not-pass in 19% of basic-tier groups, 34% of hard-tier groups and 56% of Zig groups. Basic tasks now get two runs; hard and Zig keep three, which saves about an hour per model. The bigger saving is a stop rule calibrated on 16 models: Zig, up to 12 GPU hours, runs only if the model passed at least 2 of its 9 hard runs or had at least 2 hard runs killed at the time limit.

Three is the smallest count that breaks a tie, and I had applied it to every tier alike without checking where the ties were. So I checked.

Disagreement by tier: 19, 34 and 56 percent#

The result files hold 739 three-run groups: one model, one task, three runs, and the question is whether the three agree on pass or not-pass.

In 568 groups (77%) all three runs agreed. In the other 171 (23%), one run went the other way. That overall number is mostly the basic tier talking, since it holds 567 of the groups: there, 19% are split. The hard tier splits 34% of its 98 groups. Zig splits 56% of its 45.

The ordering is the uncomfortable part. Noise grows with the tier, and so does the bill: a basic task is a small script, a Zig run can take four hours. Three runs bought the least where they were cheapest and the most where they were dearest.

What two runs would have shown#

Two runs look like a natural cut, so I replayed the groups as if run 3 had never happened.

In 68 groups (9% of all of them), runs 1 and 2 agreed and run 3 didn't. A two-run bench would have filed those as unanimous and never seen the disagreement. In the remaining 103 split groups, runs 1 and 2 already disagreed, so a two-run bench would have recorded a 1-1 split there: a draw, with nothing to decide it.

At the level of a whole arm (one model, one tier), two runs scaled by 1.5 do a decent impression of three. The median difference is 0 passes per task, the 90th percentile 0.33, and the worst 0.67 per task. On the hard tier, with three tasks, that worst case is 2 passes out of 9, which is enough to move a model across a line I care about.

The cut: basic drops to two runs, hard and Zig keep three#

So the basic tier goes to two runs: 12 cells per model instead of 18, on the tier that agrees with itself four times out of five. Hard and Zig keep three. A two-run Zig would often end 1-1: up to eight GPU hours for a result that decides nothing.

The saving is about an hour per model, out of a day and a half. The tiers with the noise are the tiers with the hours, so rerun counts alone can't make the bench much cheaper. Skipping a tier entirely can.

Stopping early: the hard tier predicts Zig, the basic tier predicts nothing#

My other bench, on the A9 (a 128 GB AMD Strix Halo box), had already proposed early-stop rules: run a cheap stage, and stop a candidate that does badly there. Two reviews of that proposal added two conditions. A time-limit kill doesn't count as a failure for the stop decision. And a stop needs one more round to confirm it, because one model's single round would have tripped the bar by luck: between two rounds, its pass/fail flipped on 4 of 10 cells.

Before adopting anything like it here, I checked which tier predicts Zig, using the 16 past models that have all three tiers:

  • All 6 models that passed at most 1 of 9 hard runs got 0 of 3 on Zig.
  • Every Zig pass came from a model that had passed at least 3 of 9 hard runs.
  • The basic tier predicts nothing. One model scored 18 of 18 on basic and 0 of 3 on Zig.

The basic tier is six small scripts; the hard tier and Zig are both ledger services graded by hidden tests. The tier that looks like Zig is the one that predicts it. Part 14 showed what the hard tier sorts on.

The rule: Zig runs only if the model passed at least 2 of 9 hard runs, or had at least 2 time-limit kills. The kill clause is the first review point carried over. Zig gets up to four hours per run, for reasons part 4 covers, so a stop saves up to 12 GPU hours per model.

One more clause: if result rows are missing, Zig is skipped and the model is logged as "missing", never as 0 PASS. A crashed run must not look like a bad model.

Before the counter ran on a new model, I ran it on models with known results. The 0 of 9 model stops. The 4 of 9 models continue. A stop rule that can't stop and a stop rule that stops everything look the same until you feed them a known answer.

A rerun that reached the changed variable#

A stop rule cuts runs; the other way to spend GPU time badly is a rerun that changes a setting no run ever reached.

A 74B MoE failed hard tasks at 64k context with a 16384-token output cap. Most of its failing hard and Zig cells ended at the cap. I checked each one in the harness's own SQLite message log: output plus reasoning tokens at or above the cap, long real reasoning, no repetition. That reads like a cap problem, so the rerun went to 128k context and a 32768 cap.

Hard went from 4 of 9 to 5 of 9. Zig stayed at 0 of 3.

Before crediting the cap with the extra pass, or clearing it, I checked whether the rerun used the room. It did: 6 of the 9 hard cells produced a reply longer than the old cap, the longest 30,487 tokens. One Zig cell hit the new cap once, again on real reasoning. So the cap was real, the rerun reached past it, and the rerun gained one pass out of nine, on a tier where a third of three-run groups split on their own. The cap was a limit; it wasn't the bottleneck. The model was dropped.

The A9 bench made the same check in the other direction: in Seven FAILs, one setting, four of the seven cells rerun with a larger context never needed the extra room. A rerun that never reaches the changed variable is a resample, whatever it scores.

Next: someone else's config, one round first#

The next test copies a public benchmarker's recommended settings for a 27B model and runs them against mine on the same weights and build. It uses the confirm round from the A9 review: one round of 9 cells first, a second only if the first comes in low, and a "degraded" call only if both come in low. Results are a couple of days out and will get their own post.

Filed under: local-llm, llm, ai-agents, benchmarks, opencode

Sources#

  • My own result files from the 3090 agent bench (local, not published): the 739 three-run groups, the 16-model calibration and the 74B MoE rerun rows.
  • Seven FAILs, one setting — the A9 bench's context rerun, for the comparison in the last section.