105 identical commands and other ways local agents fail

2026-10-31 · 13 min read

Contents

Assumes you've driven a coding agent through a client like OpenCode. Part 1 introduced the bench; part 23 has the results by task.

GLM-4.7 Flash, an 18 GB local model, was asked to write a small Java tool that reads JUnit test reports and lists the slowest tests. Somewhere in the run it typed this shell command:

rm -rf test-reports && mkdir test-reports && java SlowTests --json test-reports 2>&1

The command empties a directory and then asks the tool to summarise it. After a line from the JVM about its options, the tool answered with a JSON summary that began {"total":0. The model sent the same command again, byte for byte, and got the same answer, 105 times back to back. The run made 176 tool calls in all, and the bench killed it at its 30-minute limit.

My bench runs local models on one RTX 3090 as coding agents, through agent clients such as OpenCode: each run gets a task, a workspace and a time limit, and hidden checks the agent never sees grade what it leaves behind. A run PASSes when every check passes, is PARTIAL when only a detail check fails, and FAILs when a core check fails. Part 23 says which models passed. This part is about what the agents did on the way, read from their transcripts: how long they thought, which tools they called, what went wrong and how the failing runs differed from the passing ones. Five patterns cover most of it: replies that ran out the clock, loops, failed edits, mistyped paths, and scripts that never got written.

What the transcripts cover: 1,655 of 3,088 runs#

The result files hold 3,088 rows, one per agent run. I scanned the transcripts behind 1,655 of them (54%): 1,082 runs in OpenCode, each with its own message database, and 573 in Pi, another agent client, which writes a session file per run. Coverage per model runs from 20% to 100%. OpenCode runs before 15 October shared one database and aren't covered, and neither are the runs in two other clients.

Some numbers below come from all 3,088 rows (verdicts, kills, which hidden checks failed) and some only from the 1,655 scanned runs (anything about replies and tool calls). Each one says which. Three definitions matter:

Thinking share is the share of a run's characters that went to reasoning, against visible text and tool arguments. llama.cpp didn't report reasoning tokens separately in almost any of these runs, so it's characters, not tokens. A model served with reasoning off scores 0.
Longest reply is the largest single model reply in a run, in output tokens, thinking included.
Tool error is a call the client itself marked as failed: an edit whose text wasn't found, a path the sandbox blocked, a call that didn't fit the tool's schema. A shell command that exits non-zero is not a tool error here, because OpenCode records it as completed and Pi doesn't, so counting it would compare the clients rather than the models.

Killed runs wrote replies four times as long#

A run that reaches its time limit is killed and graded on what it left in the workspace. Over all 3,088 rows, 135 runs were killed and none of them passed. That isn't the same as broken code: 45 of the 135 left code that passed every hidden core check, and lost only on time.

In the scanned runs, the killed ones look different from the passing ones well before the limit (medians per group):

Group	Runs	Thinking share	Longest reply (tokens)	Tool calls
Killed at the time limit	61	0.81	14,613	40
PASS	1,172	0.51	3,623	14
FAIL, not killed	268	0.58	5,038	50
PARTIAL	154	0.57	8,248	58

The median passing run made 14 tool calls, its longest reply was about 3,600 tokens, and about half its characters were thinking. For killed runs the medians are 40 calls, a 14,600-token longest reply and four-fifths thinking.

Part of that gap is which models got killed: a few models collected most of the kills. Nanbeige accounts for 15 of the 61 scanned kills, and the two Granite 4.2 models for 22 more. Without the Granites, the 39 remaining kills still show a thinking share of 0.79 and a longest reply of 16,599 tokens; without Nanbeige as well, 24 kills at 0.74 and 16,570 tokens. Inside single models the thinking-share gap holds where there are enough runs to look: the qwen38pi fine-tune of Qwen3.8 27B had a share of 0.77 in its 3 killed runs against 0.42 in its 76 passes, and Bonsai 2 27B 0.80 in 6 kills against 0.74 in 29 passes, a smaller gap from a model that thinks a lot either way. The longest-reply gap doesn't hold everywhere: Granite 4.2 8B's 11 kills had shorter longest replies than its passes, 4,199 tokens against 4,891.

None of this says more thinking would have helped. When part 16 turned reasoning effort up on Qwen3.8 27B, it passed fewer audits and fewer coding runs. The client moves the number too: part 12 found the same model reasoning 3.6 times as much in OpenCode as in Pi. In these transcripts, a very long reply marks a run in trouble; I found nothing in them that shows one getting a run out of it.

Nanbeige thought until the clock ran out#

Nanbeige4.3 33B (a 3-bit quant) is the extreme case: 15 of its 21 scanned runs were killed, more than any other model. All 21 were the six small command-line tasks, each with a 30-minute limit.

Its runs came in arms, batches of runs under one set of settings. In the two arms without a thinking budget, 8 of the 9 scanned runs were killed and one passed. In one of them, the CSV summary task, the model sent 3 replies and made 3 tool calls in 30 minutes. The longest reply was 29,357 tokens, and 96% of the run's characters were reasoning. At the 20 to 22 tokens per second its other two arms measured on this card, a reply that long takes more than 20 minutes to generate. Two runs of the Java test-report task never made a tool call at all: two replies each, the longer one reaching the 32,000-token output cap, and then the clock.

Two more arms capped its reasoning with a thinking budget. The longest replies came down to between 8,196 and 11,785 tokens. None of the 12 runs passed: 7 were still killed at the limit, 4 ended on their own after 3 tool calls or fewer, and 1 ended on its own with a failing result. Shorter replies didn't turn into finished scripts; one pass in 9 became none in 12.

Output caps cut the same kind of run from the other side. In the scanned set, 68 runs had at least one reply that reached the client's output cap (or came within 2% of it), and 12 of those still passed; 26 ended on the capped reply. Bolt 74B ended 6 of its 42 runs that way. On the Spring Boot task it stopped after 317 seconds of a one-hour limit, its last reply at the 16,384-token cap, in a run whose characters were 93% thinking. Part 21 reran it with twice the cap and twice the context; it gained one hard-tier pass in nine.

Loops: 24 runs, none of them passed#

A loop here is the same tool called with byte-identical arguments at least three times back to back. That's strict: a repeat with one changed flag, or two commands alternating, doesn't count, so the real number is higher. 24 scanned runs have one. 22 failed, 2 were PARTIAL, none passed, and 8 were killed.

GLM-4.7 Flash's 105 test-report commands are the longest. The others look much the same:

gemma4-agentic-v2, a 7.4 GB Gemma 4 build, on the inbox-sorting Bash task: one ls -la ... | grep .js || echo 'no js' 87 times in a row, inside a run of 247 tool calls in 1,545 seconds. The pattern .js also matches json, so every answer was the same listing: a json folder and a .json file. The run ended on its own and failed. 12 of its 18 scanned runs contain a loop.
Granite 4.2 30B on the Go branch-age report: a cd into the test repository followed by two git branch --contains <sha> counts, 55 times in a row, each answer 0 and 1. 138 tool calls, FAIL.

In each case the answer didn't change and the model asked again. All three were long runs, 138 to 247 tool calls, and the one repeated command took between a third and three-fifths of them.

Failed edits: the biggest group of tool errors#

The scanned runs made 65,520 tool calls, and the clients marked 1,543 of them as errors, about 2.4%. Failed edits are the biggest group, in three kinds:

Old text not found, about 595 (39% of all errors). OpenCode's edit tool replaces one exact string with another, and the call fails unless the old string is in the file as given, "including whitespace, indentation, and line endings", as the error message puts it. Pi's edit tool fails the same way.
Nothing to change, about 161. The old and new text were identical, and the client rejected the edit. This isn't one model's quirk: in OpenCode alone, gemma4-agentic-v2 did it 53 times, Ornith 1.5 35B 21, GLM-4.7 Flash 12 and K2 Horizon 11.
Malformed calls. Edit and write calls that didn't fit the tool's schema, such as a write with no file path.

Most models make a few of these and recover. gemma4-agentic-v2 didn't: 464 of its 1,580 tool calls failed, 29%. In OpenCode's own error messages, 141 were edits whose old text wasn't there, 114 were edit or write calls that didn't fit the schema, and 53 were edits that changed nothing. In the inbox-sorting run above, 65 of its edits failed: 55 malformed, 10 on missing text.

Paths the model couldn't spell#

Each run's sandbox allows the workspace and refuses paths outside it. Plain refusals are the second biggest error class, 227 of the 1,543. For two models the scan split out a pattern of their own.

Flint 40B mistyped its own workspace path. In the scanned runs 251 of its 1,442 tool calls failed, 17%, and at least 174 of the failures were a misspelt copy of the path: -harness for -harnesses, runs/hb11 for runs-hb11. In its worst run, the Go ledger task, 83 of 115 calls failed in the scan's count. Part 19 traced that run call by call and checked the misspelt paths against the model's own tokenizer. Even so, 73% of Flint's scanned runs passed; part 19 found that most of its hard-tier sessions mistyped a dozen calls or fewer, and three locked in.

LFM2.5 24B-A2B dropped the last folder instead. 38 of its 44 tool errors went to the folder one level above the workspace, which the sandbox blocks: 35 of them a cd, 3 a write or an edit.

What the hidden tests caught on the ledgers#

Three hard tasks share one spec: add a money transfer to an existing ledger codebase, in Java, Go, and Zig. A fourth asks for a bug fix and a registration endpoint in a Spring Boot app. Over all rows that failed one of these on a named group of hidden tests, these groups failed most (a run can fail several):

Task	Failed runs	Most-failed groups (runs)
Java ledger	55	atomic 49, idempotency 35, concurrency 10
Go ledger	37	atomic 31, idempotency 29, concurrency 10
Zig ledger	78	idempotency 55, concurrency 47, atomic 47
Spring Boot endpoint	34	validation 28, create 8, conflict 8

In plain words:

Atomic: when storage fails partway through a transfer, for example on saving the destination account, both balances must stay as they were. 49 of 55 failed Java runs and 31 of 37 failed Go runs broke this. Part 14 followed that one rule through the hard tier.
Idempotency: a retried request with the same key must move the money once, including a retry after a failed save.
Concurrency: parallel transfers, including two in opposite directions, must neither deadlock nor lose an update. It fails most often on the Zig version.
Validation: the Spring endpoint must reject bad input with a 400 response that lists the field errors. Separately, 92 Spring rows lost a detail check that wants exactly one error entry per invalid field.

The groups for plain business rules (a basic transfer, insufficient funds, a currency mismatch) failed in 4 to 21 runs per ledger task. The groups that test what happens when something goes wrong in the middle, a failed save, a retry, a concurrent transfer, failed in up to 55.

Scripts that never got written#

The simplest failure has no code to grade. On three of the six small command-line tasks, the most common single failure reason over all rows is that the file the task asks for doesn't exist: no sortfiles.sh 69 times, no csvstat.py 51 times, no merge.js 37 times. These counts include the first weeks, when every reply was cut at 8,192 tokens and reasoning models often stopped mid-thought.

By model, Granite 4.2 30B never wrote the deliverable in 26 runs, Qwen3.6 35B in 16 and Granite 4.2 8B in 14. My model notes from a 6 October retest describe Nail 35B on the inbox task in OpenCode in one line: it explores, then stops without writing the script, 3 runs of 3. Nanbeige's killed runs are in this list too: no csvstat.py, time limit reached.

A run like that can look busy. Reading files, listing directories and planning all produce tool calls and transcript; none of it produces the one file the scorer looks for.

The boring ones: Swift 1.5 and Qwen3.8 27B#

The two models with the most scanned runs have the dullest transcripts in the set.

Swift 1.5, a Qwen3.8 27B fine-tune, has 261 scanned runs. None was killed, none hit the output cap and none looped. The client flagged 21 of its 6,349 tool calls, 0.3%. The median run made 10 tool calls, and its longest reply was about 3,900 tokens. Qwen3.8 27B itself, 175 scanned runs: no kills, a cap hit in one run, 21 errors in 3,780 calls (0.6%), a median of 11 tool calls and a longest reply around 4,100 tokens.

Over all their rows, including runs the scan doesn't cover, Swift was killed 2 times in 334 runs and Qwen3.8 27B 14 times in 250. None of those kills is in the scanned set, so I can't say what those runs looked like.

That's the shape of a passing run on this bench: about a dozen tool calls and replies a few thousand tokens long. The failing runs in this post left it in five ways, and each one is visible in the transcript without the verdict: a reply that runs into the output cap or the time limit, the same command a third time in a row, an edit that fails, a path the sandbox refuses, a session that ends with the deliverable never written.

Filed under: local-llm, llm, ai-agents, benchmarks, opencode

Sources#

My own result files and agent transcripts from the 3090 agent bench (local, not published): 3,088 result rows; 1,655 transcripts from OpenCode's per-run message databases and Pi's session files, scanned read-only.
Earlier parts of this series, linked inline, for the incidents they cover in detail.