Flint mistyped its path and blamed the tools

Assumes you've driven a coding agent through a client like OpenCode. Part 1 introduced the bench's sandbox.

On 22 October a Go run of Flint 40B made 115 tool calls, and OpenCode refused 81 of them. The model spent the session working around what its summary called a "persistent tool-output glitch", then reported its code and tests written. The workspace diff was zero bytes.

The glitch was a typo. Seven calls in, the model started mistyping its own workspace path, one token off.

A mistyped path should be the cheapest mistake an agent makes: the shell says No such file or directory, the agent retypes. In this bench most of them never reach the shell. A permission rule refuses them first, with a message that doesn't mention the path.

Flint on the bench: 18 of 18, then 3 of 9#

Flint is Recursive's new mixture-of-experts model, 39.8B parameters with about 2.5B active per token. The model card says it is "designed to investigate codebases, fix bugs, implement features, and write tests". It ran as unsloth's UD-Q4_K_XL GGUF on mainline llama.cpp a37cc46 (the b11801 release lacks its architecture), at 128k context with the card's sampler, and the experts of 8 layers in system RAM so it fits the 3090's 24 GB. The client is OpenCode 1.18.4, the same for every model on the hard tier.

The regular tier was as clean as anything on the bench: 18 of 18, and 70 of 84 edge probes, the code-quality checks that a pass doesn't need. The hard tier is where it came apart. Three of the nine hard runs passed, all three on the Java ledger task. The Go ledger went FAIL, PARTIAL, FAIL. The Zig task passed once in three.

Some of those failures are ordinary code failures: a concurrency check in one Go run, an email validation in one Spring run. Three are not. In Go run 1 and Zig run 3 the model's code never reached the disk, and in Go run 2 its tests didn't.

A 92-character path, one token off#

Every run works in a directory with a long absolute path. With the project name taken out, it has this shape:

/home/user/work/<project>-harnesses/agent-playground/runs-hb11/flint-40b/t14-ledger-go

The real one is 92 characters. OpenCode's file tools take a path on every call, and their schemas ask for it absolute, so the model types that prefix again and again. In Go run 1 the refused calls carried these versions of it:

In Zig run 3, all 42 refusals carry one variant: runs_hz13 for runs-hz13.

Running the variants through Flint's own tokenizer (the tokenizer.json from its Hugging Face repo) makes the pattern obvious. Every variant has the same token count as the correct path (30 for the Go workspace, 31 for the Zig one), with one to three tokens swapped for a near neighbour: -h became _h or /h, and nesses became ness. The token -h appears twice in the path, in -harnesses and in -hb11, and those two tokens take most of the damage. The model never adds or drops a whole segment. It picks the sibling token.

Short sessions barely trigger it. Across the 18 regular runs, Flint made 274 tool calls containing an absolute workspace path, and one of them was mistyped. On the hard tier it is spikier than a rate suggests: most sessions mistyped a dozen calls or fewer, and three locked in. Go run 1 mistyped 88 of its 99 path-carrying calls, Go run 2 mistyped 90 of 126, Zig run 3 42 of 82.

It isn't a slow decay as the context fills, either. In both sunk runs the first wrong path came early: call 7 in Go run 1, call 4 in Zig run 3.

What OpenCode does with a path outside the workspace#

A mistyped absolute path points outside the workspace, and that is OpenCode's external_directory permission. The bench sets it per run:

"external_directory": {
  "*": "deny",
  "/tmp/**": "allow",
  "/home/user/work/<project>-harnesses/agent-playground/runs-hb11/flint-40b/t14-ledger-go/**": "allow",
  "/var/home/user/work/<project>-harnesses/agent-playground/runs-hb11/flint-40b/t14-ledger-go/**": "allow"
}

Deny, not OpenCode's default ask, is deliberate. opencode run has nobody to answer a prompt, so ask auto-rejects and ends the session. Before 7 October, 53 bench rows ended that way, on scratch files in /tmp, on the workspace reached through the /home → /var/home symlink, or on a mistyped path. Deny returns an error to the model and the run continues, which is roughly what an interactive "no" amounts to.

The error the model gets back is this, from OpenCode's source at v1.18.4:

export class DeniedError extends Schema.TaggedErrorClass<DeniedError>()("PermissionDeniedError", {
  ruleset: Schema.Any,
}) {
  override get message() {
    return `The user has specified a rule which prevents you from using this specific tool call. Here are some of the relevant rules ${JSON.stringify(this.ruleset)}`
  }
}

The caller passes the rules for that permission. The path that was refused is not in the message. What the model saw was a 1,369-character message, mostly JSON rules, and one of those rules is the allow entry for the correct workspace path. Every one of the 81 refusals in Go run 1 handed the model the right spelling.

Which calls get checked is uneven, and that matters later. read, edit and write check their file path. The bash tool checks its workdir, plus the arguments of a short list of commands: cd and its relatives, rm, cp, mv, mkdir, touch, chmod, chown, cat. So cat on a mistyped path is refused, while sed -n 1,60p on the same path runs and gets an honest No such file or directory.

One more detail in the same source: the edit tool's schema calls its parameter "The absolute path to the file to modify", and the code joins a relative path onto the workspace anyway. For the file tools, a relative path would have worked every time.

Go run 1: 81 refusals read as noise#

The first six calls use the right path. Call 7 asks for errors.go under <project>-harness. From there it gets worse. Call 10 is a single cat with six files: the first under the right path, the next five under <project>-harness/…/runs/hb11. The whole call was refused.

Call 25 is the last time the model typed the correct absolute path. In the remaining 90 calls, the ones that did any work were round trips through /tmp or commands with no path at all. A few unchecked commands (tail, sed, grep, one gofmt) ran on the mistyped path and got No such file or directory, and relative paths didn't help, because those calls carried the mistyped workdir. All 15 edit and write calls in the session were refused, and every other go vet, go test and gofmt was refused for its workdir.

The session text shows how it read the refusals. From the first one, it treats the rules block as text being injected into otherwise successful output: a display bug, an environment quirk, a transient glitch. It reads files one at a time, redirects output to /tmp and reads it back, echoes marker strings to see whether output survives. When echo and go version come back clean (no path in either) it concludes the trouble is specific to go test. Around call 70 it reassures itself that its edits went through because the edit tool had reported success. Every edit and write it had attempted by then, twelve of them, had come back refused.

Call 81 gets closest. sed says the file doesn't exist in what pwd reports as the current directory. The model considers that the path might be subtly different, a different character, and decides to test a relative path. The test call still carries the mistyped workdir and is refused, and the idea doesn't come back.

The closing message:

Current state: all code and tests written and formatted (gofmt -w . run); not yet verified — go vet ./..., go test -race ./..., and gofmt -l . have repeatedly returned output replaced by a permission-rule text block (persistent tool-output glitch in this session, affecting go test/go vet/file reads).

It also lists the files it changed, under the mistyped path. The scorer's 29 points out of 100 all went to the fixture it never touched: it builds, nothing regresses, nothing untidy was added. To be fair to it, the summary says verification didn't happen and asks how to proceed. That half is honest. The half that says the code exists is not.

Go run 2 locked in the same way, 70 refusals and 90 of 126 path-carrying calls mistyped, and got further by accident. It wrote its source files with cat > heredocs. OpenCode doesn't check a redirect target, and those heredocs carried the correct absolute path, so they landed; the one with the mistyped path ran and failed with No such file or directory. That code passed all 15 hidden core tests. The test file went the same way minus the luck: cat > transfer_test.go, a relative path in a call whose workdir was mistyped, refused. The run scored PARTIAL with no tests of its own, and its summary says "Wrote 8 new tests in transfer_test.go", with the blame again on permission-rule text replacing tool output.

Zig run 3: a real green build of the old code#

The Zig run is the stranger one, because its checks passed.

It had one typo, runs_hz13 for runs-hz13, and it landed mostly in file-tool paths: 34 of the 42 refusals are reads, edits and writes. Most of its bash calls kept the correct workdir. Call 63 edited src/errors.zig with the correct path and landed. Calls 64 to 73 edited audit.zig, statement.zig, service.zig and root.zig under runs_hz13, and all ten were refused. So were the two writes of tests/transfer_test.zig and the two edits after them, one of which wired it into the test root.

Around those, at calls 74–75 and 80–81, it ran zig build and zig build test in the right directory. Every one came back with no output or EXIT=0. They were telling the truth: the fixture plus three new error names compiles, and its existing tests pass. The model wondered for a moment whether the new tests had run, and decided they must have, because the test root imports the transfer tests. That import was in one of the refused edits.

At call 88, git status showed one modified file. The model checked: service.zig has no transfer, tests/ has no transfer test, every file shares one modification time except errors.zig. It even noted that a passing test run with the new test file would have been impossible without the feature. Then it settled on the wrong explanation:

So the green build/test results were from the edited state — but the edits may have been reverted mid-session.

The inconsistency report above that line is accurate, and more than many runs manage. The cause is wrong: nothing was reverted, nothing landed. The "Verified" line with three ticks describes checks that ran on code without the feature. The diff is 6 added lines in errors.zig, and the hidden tests scored 0 of 15.

Whose fault, and what twelve runs can say#

The paths are the bench's, but every model gets the same shape, differing only in the model's directory name. Across the same 12 hard runs, Flint drew 224 refusals. The other twelve OpenCode models drew between 0 and 11 each, refusals of any kind. The rows stand as Flint results.

The deny rule doesn't escape either. Under ask, going by the bench's own 7 October observation, Go run 1 would have ended at call 7: the same FAIL, and no summary claiming the code exists. Deny is still the right default for the sessions ask used to end on a /tmp file or the symlinked path. But it converts a typo into a refusal that names "the user" and a rule list, and leaves the model to diff 1,369 characters of JSON against its own input. Flint never did.

What this doesn't show: why the hard tier triggers it. Two of three Go sessions locked in, one of three Zig, none of the Java or Spring ones, and that is twelve runs. The tokenizer check says what the typos are, one sibling token at a time, not why the model emits them. Speed was never Flint's problem: its Zig runs took 1288 s on average against 4404 s for K2 Horizon, which passed all three.

So the cheapest mistake an agent can make cost two runs outright and the tests of a third, and in all three the report described work that was never on disk.

Filed under: local-llm, llm, ai-agents, benchmarks, opencode

Read next#

• I blamed OpenCode; a 2-bit model passed Zig in it — the same external_directory rule, and the time it looked guilty and wasn't.
• The models deleted the benchmark's log — another way agents misread what the harness put in front of them.
• MTP acceptance is a side effect, not a target — the previous part of this series.

Sources#

• Recursive Flint model card — parameter counts, intended use, sampler settings.
• unsloth/flint-GGUF — the UD-Q4_K_XL quant used here; tokenizer.json for the token check is in the base repo.
• OpenCode v1.18.4 core/src/v1/permission.ts — the DeniedError message.
• OpenCode v1.18.4 permission/index.ts — a deny passes the rules for that permission, not the requested path.
• OpenCode v1.18.4 tool/shell.ts — the commands whose path arguments are checked; the workdir check is at L626.
• OpenCode v1.18.4 tool/edit.ts — "The absolute path to the file to modify", and the relative-path join at L80-L83.
• OpenCode v1.18.4 tool/external-directory.ts — the check the read, edit and write tools run.
• Benchmark artifacts (local, not published): OpenCode session databases for all 30 Flint runs (22 October) and the hard-tier runs of 12 other models (18–22 October), result rows, workspace diffs, the bench's per-run OpenCode config and its 7 October note on deny vs ask.