My benchmark read my CLAUDE.md

Assumes you've pointed a coding agent like OpenCode at a local OpenAI-compatible server.

The payload audit on 14 October was built to answer one dull question: do the coding clients in my local agent benchmark send their own sampler settings and quietly override the server's? For the model ids it captured, they didn't. OpenCode 1.18.4 and pi both sent no temperature, no top_p, nothing, and I took that as the answer for the whole benchmark.

The same capture had a column for system prompt length. pi's was 2,744 characters. OpenCode's was 53,483, and turning off its Claude Code compatibility cut that to 10,785. The difference was my ~/.claude: the global CLAUDE.md, whole, and most of a list of 22 skills. The benchmark had never turned it off.

The audit: a fake server that only takes notes#

The tool is a 35-line Python script, fake.py. It speaks just enough of the OpenAI API to keep a client happy: /v1/models, a /props that reports a 64k context, and a chat endpoint that streams back the word "Done." and stops. Every POST body gets logged with messages and tools stripped out, replaced by counts, the tool names, and the length of the joined system messages. No GPU, no model, no waiting.

OpenCode got the exact per-run config the bench generates, with baseURL pointed at the fake. pi got the bench's own flags. The pi capture survives as one line:

{"model": "qwen38-27b-128k", "stream": true, "stream_options": {"include_usage": true}, "store": false, "max_tokens": 32768, "_n_messages": 2, "_n_tools": 4, "_path": "/v1/chat/completions", "_tools": ["read", "bash", "edit", "write"], "_sys_len": 2744}

No sampler keys. For pi, that was the question answered. The OpenCode lines were overwritten by later runs of the script, so for them only the summary table in the audit notes remains, and the numbers below come from it.

Where 53,483 characters came from#

Splitting the prompt by what was in it:

• OpenCode's own base prompt: 9,002 characters.
• My ~/.claude/CLAUDE.md: 28,504 characters, the whole file plus a short header.
• A skill list: 15,901 characters describing 22 skills.

With OPENCODE_DISABLE_CLAUDE_CODE=1 set, the same config produced 10,785. Three skills survived the flag, about 1.7k characters: OpenCode's built-in one and two that live in ~/.agents/skills, which the flag doesn't cover.

At 3.6 characters per token, the file plus the skill list, 44,405 characters, comes to about 12k tokens and the whole prompt to about 15k. Those are division, not tokenizer output. OpenCode in the bench's 64k profile compacted at a median prompt of about 35k tokens, so roughly 15k of a 35k working window was spoken for at turn one, and close to 8k of that was one personal file of gotchas.

The benchmark's tasks are things like a Zig ledger and a Spring Boot service. None of that file was written for them.

The harness isolated every client except this one#

This is the part that stings. The harness already knew about this class of leak. It had isolated three of its four clients from it:

• Claude Code: CLAUDE_CONFIG_DIR points at an empty bench-local directory, and claudeMdExcludes keeps the repository's own CLAUDE.md out.
• pi: --no-context-files --no-skills --no-extensions --no-prompt-templates --offline, tagged isolated in the row.
• omp: --no-skills, with a comment that ~/.agents/skills would otherwise load.

The Claude Code branch even explains why, in a comment that has been in run-scenario.sh since the repository's first commit:

Left alone, this cell would measure that config rather than the harness, and pi-bare/opencode have no equivalent — the comparison would be meaningless.

pi doesn't read ~/.claude; its flags exist for other files it would load. OpenCode had exactly the equivalent, and its documentation says so in a section titled "Claude Code Compatibility". For users migrating from Claude Code, it reads ~/.claude/CLAUDE.md as a fallback global rules file when there is no ~/.config/opencode/AGENTS.md, and it searches ~/.claude/skills/ for skills. Both are on unless you set an environment variable. OpenCode did what it documents. The harness had nothing in place for the client it ran most of its rows through.

One thing it did not load: the repository's own CLAUDE.md. Bench workspaces sit inside the repo, but each one is its own git repository, and OpenCode's skills docs say its project discovery walks up from the working directory only until it reaches the git worktree. For rule files the audit saw the same stop: run from an in-repo workspace, the repository's CLAUDE.md stayed out.

A confound that changed mid-study#

A fixed extra prompt would at least be a fixed condition. This one wasn't, and the obvious record of it is the wrong one. ~/.claude is a git repository, but most of its commits are made on my laptop. The bench runs on a separate GPU box, which only became a clone of that repository on the afternoon of 11 October and took each later commit only when it pulled.

The better record was on the GPU box all along. Before the fix, every bench OpenCode run shared one opencode.db, and it stores the token usage the server reported for each request. Taking the first request of every run of one task, the Zig ledger, gives a timeline in real tokens rather than estimates:

The jump of about 7k tokens lands on the first run after the GPU box's ~/.claude became a clone, at 15:56 on 11 October. The next step, a little over 1k, follows a pull at 17:28 on 13 October that took the file from 24,527 to 28,502 bytes. By 14 October the runs carried close to 12k tokens more than the isolated ones, which is about what the character estimate above said.

So the largest single change in the study's prompt came from setting up config sync between two machines. Before that, the runs were already some 2k to 4k tokens heavier than isolated ones, from whatever older configuration the GPU box had; nothing recorded what it was.

Which results it touches#

The patch comment draws the line plainly: rows from before the fix are a separate condition. In practice:

• Every OpenCode row on the audited version. The flag appears nowhere in the tree before the patch, and the compatibility is on by default. 597 OpenCode rows record version 1.18.4 on the GPU box, against 179 with the new tag so far. How much extra they carried depends on the date: roughly 2k to 4k tokens before the afternoon of 11 October, 10k to 12k after. Another 670 older OpenCode rows record no client version, some from another host; no capture exists for those.
• OpenCode against pi was never like for like. The 251 pi rows that record a client all ran with the isolation flags, so any harness comparison against them gave OpenCode the smaller working window.
• Rows from different dates differ in prompt. First requests on the same Zig task started at about 12k tokens on 10 October and about 20k on 14 October.
• Most batches shared one condition, but not all. A queue that ran entirely on one side of the sync gave every model the same prompt, so its model-against-model comparison holds. The 128k test-first Zig queue did not: in each of its three replicates swift15 ran on the afternoon of 11 October, before the sync, at about 12.1k tokens, and the other three models ran after 16:04 at about 19.2k. One 64k queue also straddles the 13 October pull, at about 19.2k against 20.3k.
• The Zig context result is affected most. At 64k a working window of about 35k is where a prompt that grew by 7k in an afternoon hurts, and the arms behind Zig needed context, not time ran on both sides of that step.

What I can't give is the effect size. No rerun isolated it: from the same day, bench rows for the four Qwen3.8-27B models also run with MTP speculative decoding on by default. For those models, old and new rows differ in two variables at once, so "how much did the extra prompt cost" has no measured answer here. It's a confound with a known direction and an unknown size.

The fix: one variable, a private data dir and a tag#

The isolation patch, applied just after midnight on 15 October, adds two exports to the OpenCode branch and makes the condition visible in every row:

export OPENCODE_DISABLE_CLAUDE_CODE=1 XDG_DATA_HOME="$META/xdg-data"
mkdir -p "$XDG_DATA_HOME"
CLIENT="opencode $(opencode --version 2>/dev/null | head -1) extdir=deny+tmp noclaude xdg=run"

The private XDG_DATA_HOME also gives each run its own opencode.db instead of the one shared with my interactive sessions, and tool calls are now counted from that database rather than grepped from a log the model can delete. Run logs moved out of the workspace, and each row now records the serving engine build, the weights with their SHA-256, and the server's default sampler. Smoke runs of both OpenCode and pi passed that night with the new row fields.

Re-running the capture two days later, on OpenCode 1.18.33 and a CLAUDE.md that had grown again, gives the same prompt picture. These are the main requests from each log, as tail -n1 prints them:

==> oc-default.jsonl <==
{"model": "qwen36-27b", "max_tokens": 16384, "tool_choice": "auto", "stream": true, "stream_options": {"include_usage": true}, "_n_messages": 2, "_n_tools": 9, "_path": "/v1/chat/completions", "_tools": ["bash", "edit", "glob", "grep", "read", "skill", "task", "todowrite", "write"], "_sys_len": 54191}

==> oc-noclaude.jsonl <==
{"model": "qwen36-27b", "max_tokens": 16384, "tool_choice": "auto", "stream": true, "stream_options": {"include_usage": true}, "_n_messages": 2, "_n_tools": 9, "_path": "/v1/chat/completions", "_tools": ["bash", "edit", "glob", "grep", "read", "skill", "task", "todowrite", "write"], "_sys_len": 10771}

54,191 characters without the flag, 10,771 with it. The 29,032-character version in force that morning is in there whole, next to a 22-entry skill list.

Three smaller things the audit turned up#

• max_tokens was 32000, not 32768. The bench config asked OpenCode for a 32,768-token output limit; it sent 32,000. The 1.18.33 binary carries a built-in OUTPUT_TOKEN_MAX of 32000, with an OPENCODE_EXPERIMENTAL_OUTPUT_TOKEN_MAX override. I read that constant from the binary, not the source, so the exact rule is my inference. It matches the compaction data, though: the smallest prompt at which a 64k run compacted was 33,541 tokens, and 65,536 − 32,000 is 33,536.
• One extra request per run. Before the real work, OpenCode sends a title-generation request: three messages, a 2,096-character system prompt, no tools. Harmless, but it is a request a server-side counter will see.

The third was a hang: opencode run with stdin left as an open pipe waits forever at plugin init. Bench runs under systemd get /dev/null and were never affected; the patch adds < /dev/null anyway. It came back while the notes for this post were being written. An unquoted heredoc expanded a backticked opencode run in the text, the shell executed it, and it sat there at startup, sending nothing, until it was killed.

Every client in the bench had been isolated after somebody thought about what it might load. The one that ran most of the rows had been isolated by a comment saying it didn't need to be.

Filed under: local-llm, llm, ai-agents, benchmarks, opencode

Read next#

• Zig needed context, not time — the result this confound touched most.
• I blamed OpenCode; a 2-bit model passed Zig in it — the experiment that cleared OpenCode of the Zig failures, also run before the fix.
• The models deleted the benchmark's log — the other half of the same harness fix: the run log the models could see.

Sources#

• OpenCode docs: Rules, Claude Code Compatibility — ~/.claude/CLAUDE.md as a fallback global rules file, ~/.claude/skills/, and the OPENCODE_DISABLE_CLAUDE_CODE, _PROMPT and _SKILLS variables.
• OpenCode docs: Agent Skills — skill search paths, including ~/.claude/skills and ~/.agents/skills, and project discovery stopping at the git worktree.
• Benchmark artifacts (local, not published): the 2026-10-14 payload capture notes and fake.py, the pi capture line, bench/patches/harness-isolation.patch and run-scenario.sh before and after it, the results files, and a 2026-10-16 re-run of the capture against OpenCode 1.18.33.
• ~/.claude git history: per-commit byte sizes of CLAUDE.md (git cat-file -s), 24 September to 14 October 2026.