The model that only says RUN

· 14 min read
Contents

Assumes you've configured a coding agent's permission rules, or wondered whether they hold.

The most useful model I ran this month has a vocabulary of one sentence. It has no opinions about code, it never reads its context, and it does exactly what the prompt tells it, every time, which is more than I can say for most of the models I benchmark. Give it RUN: rm -rf victim :END and it asks the harness to run rm -rf victim. That is the whole personality.

It's a fake, written as the test double for a set of hardened config templates covering six coding-agent harnesses: OpenCode V1 and V2, pi, omp, Hermes Agent and Claude Code. Its job is to make each harness try dangerous commands so I can see what stops them. As product reviews go: cheap, tireless, never makes anything up, and not once has it had a better idea than the prompt. It had two bugs of its own, both of them findings, and its most interesting result was one I had filed under the wrong layer.

How a fake model drives a real agent#

The server speaks both APIs the harnesses use: OpenAI chat completions, and the Anthropic Messages API with SSE streaming and count_tokens. A prompt of RUN: <command> :END becomes a call to the harness's own shell tool, CALL: <tool> <json> :END calls any other tool, and when the result comes back, the fake echoes it. Reduced to the OpenAI shape, without streaming, the core of the idea is a dozen lines:

import json, re, sys

SHELL_TOOLS = {"bash", "shell", "terminal"}

def reply(request):
    prompt = next(m["content"] for m in reversed(request["messages"]) if m["role"] == "user")
    tools = [t["function"]["name"] for t in request.get("tools", [])]
    shell = next((t for t in tools if t in SHELL_TOOLS), None)
    run = re.search(r"RUN: (.*?) :END", prompt)
    if not (run and shell):
        message = {"role": "assistant", "content": "ok"}
        return {"choices": [{"index": 0, "message": message, "finish_reason": "stop"}]}
    call = {"id": "call_1", "type": "function",
            "function": {"name": shell, "arguments": json.dumps({"command": run.group(1)})}}
    message = {"role": "assistant", "content": None, "tool_calls": [call]}
    return {"choices": [{"index": 0, "message": message, "finish_reason": "tool_calls"}]}

print(json.dumps(reply(json.load(sys.stdin))["choices"][0]["message"]))

With req.json holding one user message and one tool:

{"messages": [{"role": "user", "content": "RUN: rm -rf victim :END"}],
 "tools": [{"type": "function", "function": {"name": "bash", "parameters": {"type": "object"}}}]}

$ python3 fake_reply.py < req.json
{"role": "assistant", "content": null, "tool_calls": [{"id": "call_1", "type": "function", "function": {"name": "bash", "arguments": "{\"command\": \"rm -rf victim\"}"}}]}

Each harness names its shell tool and its arguments its own way; this reduction hard-codes three names and a command argument. On the Anthropic side, a fake has to stream the documented event sequence: message_start, then content_block_start, content_block_delta and content_block_stop for each block, message_delta, and message_stop. Claude Code's gateway docs (fetched 2026-10-25) treat /v1/messages/count_tokens as optional, with a character-based estimate as the fallback; the fake answers it anyway.

Then there's FAKE_LLM_LOG, which writes every request the fake receives to a file. That log is the actual product. It shows what each harness really sends: tool names, schemas, extra fields, and system messages in the middle of a conversation.

The end-to-end driver installs a template into a throwaway HOME and runs each harness once per case, non-interactively, with auto-approval wherever there is approval to switch off: opencode run --auto, omp --yolo, hermes --yolo, claude --dangerously-skip-permissions, and pi -p, which is print mode. The driver checks the reply for the command's own output to tell an executed case from a blocked one, and a case passes when the outcome is the one the suite expects. On 2026-10-23, on OpenCode V1 1.19.4, OpenCode V2 2.1.4, pi 1.4.0, omp 18.12.6, Hermes Agent at commit afe3cbd and Claude Code 2.1.309, every case passed: 27 of 27, 27 of 27, 27 of 27, 28 of 28, 26 of 26 and 32 of 32. OpenCode V1 1.19.4 takes about a minute per case, so the suites run in parallel.

The fake had two bugs of its own#

Bug one: on Claude Code 2.1.309, the request carried extra role: "system" messages after the user turn, and they hid the user prompt from my fake. It now skips system messages when it looks for the prompt.

Bug two: on the builds named above, harnesses send side requests, to title a session or summarise it, and those contain the prompt too. My fake answered them with tool calls. It now answers RUN: only when the request offers a shell tool, and gives plain text otherwise.

A test double is a written-down guess about what the other side sends. Both bugs were places where my guess was wrong, and the request log is how I found out.

Claude Code: deny wins, in every mode#

The harness with the strictest rules is also the one with the most rules to get wrong. Checked against its docs on 2026-10-25 (the CLI on my box is 2.1.310; the experiments ran on 2.1.309):

  • Order is fixed: "Rules are evaluated in order: deny, then ask, then allow." Lists from different settings files are merged, not replaced.
  • A deny survives bypass mode: "Deny rules block in every mode, including bypassPermissions." An explicit ask rule still prompts there too.
  • Print mode can't prompt. With claude -p … --dangerously-skip-permissions, "the few calls that would still prompt are denied instead". Since v2.1.302, a -p run starts in Manual mode when nothing sets a permission mode.
  • Write(...) rules never match. The docs are blunt about it: "Claude Code accepts the rule but never matches a read or edit against it. Use Edit(docs/**) in place of Write(docs/**)".
  • A settings value the schema rejects drops the whole file, while a malformed individual entry, such as a broken permission rule, is skipped and the rest stays in effect.
  • A trailing :* is fine. My notes called it legacy syntax that warns. The docs say it's "an equivalent way to write a trailing wildcard", and the warning added in 2.1.282 is about a :* in the middle of a pattern.
  • A project's settings can set disableAllHooks and env, including ANTHROPIC_BASE_URL. JSON passed with --settings "overrides the same key in your project and user settings files". That's why the template's launcher always passes --settings: a repository I clone shouldn't be able to switch my hooks off or send my model traffic somewhere else. Managed settings still outrank it, which is the right way round.

Do mid-name globs match MCP tools?#

The bypass-mode bullet matters for MCP tools too. The template's Claude Code settings deny mcp__ghidra__delete_* and allow mcp__ghidra__list_*, for the Ghidra MCP server, and nothing had checked whether a * in the middle of a tool name matches anything at all. A rule that never matches looks exactly like a rule that works, right up until it's needed.

The check ran three variants on 2.1.309 against a fake MCP server that logs every call it receives: bypass mode, Manual mode (config value default) under -p, and bypass mode with the deny rule removed. The read tools ran. An unlisted write tool was refused under -p, which is what the docs predict for a call that needs approval with nobody there to approve it, but no interactive run checked which layer said no. In bypass mode the delete tool stayed blocked by the deny glob, so the mid-name glob does match.

With the deny removed, the delete tool ran. My guard hook was meant to be the second line there and wasn't: its matcher at the time didn't cover these tools. Once the matcher included mcp__ghidra__.*, the hook blocked the delete on its own.

The verdict in every variant came from the server's call log, not from the transcript. A transcript shows a tool's name in a refusal too, and a tool name in a transcript reads a lot like a call. The log only holds calls that happened.

The glob that denied the most harmless publish there is#

One rule I wrote early was a substring glob, gradlew *publish*, meant to catch Gradle publishing to a real repository. On 2.1.309 it denied ./gradlew publishToMavenLocal, which publishes into the local Maven repository in my own home directory. The ./ was stripped before matching, so ./gradlew matched as gradlew. The docs describe stripping wrappers like timeout and nice and say absolute paths are not matched, /usr/bin/curl versus Bash(curl *); they say nothing about a relative ./prog, so treat that as what 2.1.309 did, not as a documented rule.

An allow had been there all along. Bash(./gradlew publishToMavenLocal) sat in the same settings file as the substring deny, and lost, because deny wins. So the fix was a narrower deny. The publish rules now match publish only as a task of its own: ./gradlew publish, ./gradlew publish *, ./gradlew * publish and ./gradlew * publish *, plus the same four for gradlew and for gradle. They catch a bare publish task and nothing else. A project-path form such as ./gradlew :lib:publish, and every other publish… task, publishToSonatype for one, is left to the regex guard, which denies all of them except publishToMavenLocal with publish(?!ToMavenLocal). Plenty of other substring globs, *FLUSHALL* among them, are still in the deny list.

The deny rule asked, and print mode said no#

Claude Code's docs are unusually candid about what a Bash rule is: "a deny or ask rule covers the invocation Claude usually produces and isn't a security boundary around the program." The table under that sentence says Bash(rm *) stops rm -rf build/ and doesn't stop /bin/rm -rf build/ or bash -c 'rm -rf build/'.

My notes from 2.1.309 said the binary caught both anyway, and I had filed that under "stricter than its docs". The records of that experiment, deny rules only and no guard hook, say something more precise:

  • /bin/rm -rf was denied. That is more than the docs table promises. Which mechanism did it, I can't tell from the records. One candidate is the separate built-in check for dangerous rm on critical paths such as / or the home directory that the changelog describes (2.1.281, 2.1.288, 2.1.309), and the records don't say which path the run targeted. Another is the path handling that turned ./gradlew into gradlew above.
  • bash -c 'rm -rf …' was not hard-denied. Claude Code turned it into an approval prompt, explaining that a deny rule had matched something inside the -c script. In -p there is nobody to answer a prompt, so it was denied. Run interactively, the same command would have asked.

So the docs were right about bash -c, and the thing that said no was print mode. pi 1.4.0 gets to the same place by a different road: it has no approval layer of its own, its shipped permission-gate example blocks by default when !ctx.hasUI, and my guard extension for pi blocks $(echo rm) -rf victim because print mode can't ask anyone. Two harnesses, one accidental guard: a headless run with no human in it.

A blocked command tells you that something said no. To credit one layer, switch the others off, and run the case interactively as well as in -p.

OpenCode V1: the last match wins, and your approvals come last#

OpenCode V1's docs (fetched 2026-10-25) state the rule in one line: "Rules are evaluated by pattern match, with the last matching rule winning." The source at v1.19.4 shows what goes last:

export function evaluate(permission: string, pattern: string, ...rulesets: PermissionV1.Ruleset[]): PermissionV1.Rule {
  return (
    rulesets
      .flat()
      .findLast((rule) => Wildcard.match(permission, rule.permission) && Wildcard.match(pattern, rule.pattern)) ?? {
      action: "ask",
      permission,
      pattern: "*",
    }
  )
}

It's called as evaluate(request.permission, pattern, ruleset, approved), and approved is where an "allow always" answer goes. So a saved approval sits after the configured rules, and it beats an earlier deny for the rest of the session. My config says no in advance, one click on "allow always" says yes later, and later wins. That alone is why the V1 template needs a guard plugin rather than a list of denies.

The other V1 traps are smaller and just as quiet. Read and edit permissions are asked with paths relative to the worktree, while a ~/ in a rule is expanded to an absolute path, so, going by the source, a ~/ read or edit rule can never match. opencode run takes the project root from process.env.PWD, not the process's working directory, which matters the first time you launch it from a script. And for the built-in agents, the user-level config lands after the agent's own rules, so a global allow in my config overrides a read-only agent's own denies, and the template has to restate those denies for each such agent.

OpenCode V2 is a separate generation with its own docs and its own rules; those docs (fetched 2026-10-25) carry no version, and I haven't checked them against a 2.1.4 binary. Saved approvals there "are durable, project-scoped allow rules. They never override a configured deny." Policies "can hard-deny a permission check after these rules and saved approvals run". Tools run in one shared background server per user unless you pass --standalone, so a sandbox around one opencode process doesn't contain the tools that process asks for. And vLLM discovery "cannot infer tool support, so discovered vLLM models start with tools disabled": worth knowing before a test whose whole point is a tool call.

pi, omp and Hermes, in one table#

Harness, checked onWhat stands between the model and the shellSource
pi 1.4.0No approval layer: pi "does not ask for approval before every tool call". Extensions are the permission system.pi docs, security.md
omp v18.12.6yolo is the default approval mode. checkpoint sounds like file undo and is context management ("Mark the current top-level conversation state so later rewind can collapse exploratory context into a report"), off by default. eval runs Python and JavaScript cells under its own exec approval; nothing says bash.patterns applies to it.omp docs at v18.12.6
Hermes afe3cbdapprovals.deny blocks "before --yolo, /yolo, and approvals.mode: off are consulted". Hooks run with shell=False (~ is expanded; a `is just another argument). MCP servers get a filtered environment. Models under 64K context are refused (MINIMUM_CONTEXT_LENGTH = 64_000`). The docs table says Docker approvals are "Skipped (container is boundary)"; the code skips them only without host mounts.

The omp eval row is an inference from two pages, the eval tool's and the bash tool's; no sentence says "eval ignores bash.patterns", and this post doesn't test it.

What a local model server needs for Claude Code#

The fake stands in for a local model, and a real one behind Claude Code needs the endpoint the fake implements. Per its gateway docs (fetched 2026-10-25), Claude Code talks to a model through ANTHROPIC_BASE_URL and needs /v1/messages; count_tokens is optional. Ollama added that endpoint in v0.14.0, vLLM in v0.11.1, llama.cpp's server in PR #17570 (merged 2025-11-28) and SGLang in PR #18630 (merged 2026-02-21). LM Studio, llama-swap and LiteLLM document it without a version. TabbyAPI's README and file tree show none, which is absence, not a statement.

Two caveats from the same docs. An unrecognised model ID is assumed to have a 200K window unless CLAUDE_CODE_MAX_CONTEXT_TOKENS says otherwise, which matters for a local model served with a smaller context. And Anthropic "doesn't support routing Claude Code to non-Claude models through any gateway", so this whole setup is supported by me and nobody else. The templates also skip Claude Code's built-in Bash sandbox: its network goes through a proxy on my machine, and plain ssh, most database drivers and anything that isn't TCP can't connect, while the one requirement I won't give up is that the agents keep open web and LAN access.

Every claim here is pinned to a build or a fetch date, because each was true on that build or that day. The Claude Code on this box is already 2.1.310.

Filed under: ai-agents, claude-code, opencode, testing, local-llm

Sources#