The models deleted the benchmark's log

2026-10-16 · 7 min read

Contents

Assumes you've run a coding agent inside a benchmark harness, or written one.

Tool call 225 of a two-hour Zig run was two words long.

rm run-1.log

That was the last thing qwen38pi, a 27B local model driven through OpenCode, did on the benchmark's ledger task. The task never mentioned the file. run-1.log was the harness's live log of that same session. It was also the only thing the harness read to count the session's tool calls. The result row says PARTIAL, 8,131 seconds, toolcalls: 0. (It says PASS now: a 22 October rescore stopped counting the Mermaid-diagram check toward the verdict, the one detail that run had missed.) OpenCode's own database holds 225 tool parts for the session.

Where run-1.log lived: inside the agent's workspace#

The harness started each agent from its workspace directory and redirected the output into the current directory:

timeout "$TMO" opencode "${ARGS[@]}" "$(cat "$P")" > "run-$((i+1)).log" 2>&1

Untracked, uncommitted, sitting next to src/ and tests/, growing while the agent worked. To the model it was one more file in the repository it had been handed. It read like what it was, a transcript of an agent working on this exact task. The models concluded that a previous agent had been here.

An earlier audit of the OpenCode database, done for the post on why these Zig runs needed context, counted how often. In one qwen38 run the model's reasoning names run-1.log 61 times. That run compacted its context ten times. The third compaction's summary came back as raw tool-call markup with no objective in it, and soon after, the model was running this:

grep -rn "transfer\|idempot" run-1.log | head -20

Its own summary had lost the task, so it went looking in the record of its own earlier work. Twelve tool calls in that run named the file.

Three rm commands, three rows with zero tool calls#

Reading the log makes it part of what the model sees. Deleting it breaks the data. Three sessions in the shared OpenCode database deleted a harness log, all three on the Zig ledger task:

Run	Model	Command	Call # / total	Verdict
tddlong13	swift15	`rm -rf t t2 t3 t4 run-1.log && mkdir docs`	111 / 116	PASS
steerrevlong11	thinkingcap	`rm run-1.log run-2.log && git status --short`	123 / 129	PASS
tdd64long13	qwen38pi	`rm run-1.log`	225 / 225	PARTIAL (PASS since 22 October)

swift15 swept the log out along with four directories named t through t4. thinkingcap, on a steered run, also took run-2.log, the live log of the turn issuing the rm. qwen38pi had first opened the file at call 38 (head -50 run-1.log, among other things) at 01:11:36. It deleted it at 03:10:54, in its final tool call.

Here is the counter that turned those deletions into zeros:

TOOLCALLS=$(sed 's/\x1b\[[0-9;]*m//g' run-*.log 2>/dev/null | grep -cE '^(← |\$ )')

When the glob matches nothing, bash hands sed the literal string run-*.log. sed fails into /dev/null, and grep -c counts the lines of nothing. In an empty directory, with set -v echoing each command:

ls -A | wc -l
0
TOOLCALLS=$(sed 's/\x1b\[[0-9;]*m//g' run-*.log 2>/dev/null | grep -cE '^(← |\$ )')
echo "toolcalls=$TOOLCALLS"
toolcalls=0

Nothing errors. The row is valid JSON with a plausible integer in it. Across every result file, 40 rows tagged opencode record zero tool calls and more than two minutes of wall time. Thirty-seven of them are FAILs. The three that aren't are the three in the table.

The grep was wrong before it hit zero, too. On sibling runs where the log survived, it counted 79 tool calls against 99 in the database at best, and 30 against 199 at worst. It only matched lines starting with ← or $ , so some tool kinds never registered. The zero was the version of the error big enough to notice.

The answer-key scan read the same file#

The tool count wasn't the only thing reading that log. Workspaces live inside the harness repo, the hidden tests live in bench/scorers, and every agent has a shell. Reading a file leaves nothing on disk, so the harness flags PEEKED-AT-ANSWER-KEY by grepping what the agent saw:

PEEK="$( { cat "$W"/run-*.log 2>/dev/null; find "$META" -path '*-session*' -name '*.jsonl' -exec cat {} + 2>/dev/null; } \
  | grep -aoE 'bench/scorers[^" ]{0,60}|scorers/(hidden|reference|mutations|data/|gen-u|u[0-9])[^" ]{0,40}|ZzHidden[A-Za-z]*|zz_hidden_[a-z_]*|zz_runner\.zig|TestZz_[A-Za-z_]*|selftest-scorers' \
  | sort -u | head -3 | tr '\n' ' ')"

The session-JSONL half covers the pi and omp harnesses. For OpenCode the run log was the whole scan. Delete the log and the scan goes blind.

That brings us to thinkingcap on steerrevlong11. In a steered run, turn 1 is scored, the names of the failing checks go back to the model as a reviewer's list, and turn 2 gets to fix them. Turn 1 scored PARTIAL. Here is what turn 2 did, from the tool calls in the database:

Call 76 listed the workspace's parent directory. The harness keeps a t15-ledger-zig.meta directory there, next to the workspace.
Call 78, at 11:04:50, ran find on that directory.
Calls 79–84 read the turn-1 scorer report and grepped the hidden-test log.
Calls 87, 90 and 91 grepped zz_hidden_helpers.zig, the hidden suite's helper file. The scorer had copied it there when it graded turn 1. The first of those calls (path shortened):

M=".../t15-ledger-zig.meta/turn1/t15-score"; command grep -n "memoryScenario" -A 40 "$M/hidden/zz_hidden_helpers.zig" | head -60

That one ran at 11:11:14. At 11:25:05, call 123 was rm run-1.log run-2.log && git status --short. The final verdict was PASS. The row's notes field says wall from TMO_OVERRIDE; and nothing else. No PEEKED-AT-ANSWER-KEY.

Running the harness's own peek regex over that session's tool calls and their output today hits four answer-key strings. It hits nothing in the other two sessions that deleted a log. I'm not going to call it covering its tracks. The database records commands and has no column for motive. The same tidy-up shows up in two runs where the scan finds nothing.

The fix moved the logs next door#

The 2026-10-15 isolation patch moved every harness's log out of the workspace:

-    timeout "$TMO" opencode "${ARGS[@]}" "$(cat "$P")" > "run-$((i+1)).log" 2>&1
+    timeout "$TMO" opencode "${ARGS[@]}" "$(cat "$P")" < /dev/null > "$META/run-$((i+1)).log" 2>&1

Tool counts now come from a per-run opencode.db: count the parts of type tool, print -1 if the file can't be read. The peek scan reads $META/run-*.log. The workspace holds the task and nothing else.

META is defined as "$W.meta", though, a sibling of the workspace. The logs are out of the current directory, not out of reach. thinkingcap found that directory with ls, and it wasn't the only one. Eleven sessions in the shared database sent at least one tool call into a .meta directory, usually something like ls ../t15-ledger-zig.meta/ 2>/dev/null; cat ../t15-ledger-zig.meta/* 2>/dev/null | head -100. The move protects the logs from an rm in the working directory and from a glob that silently matches nothing. It doesn't hide them. Steered runs still score turn 1 into $META/turn1 while turn 2 is live, and on this task that includes the copy of the hidden suite.

The model has no way to tell which files are the task and which belong to the harness. Anything the harness writes where the agent can see it becomes part of the task. In these runs, that one log got read as a previous agent's notes, swept up as clutter, and, in the one run that read the answer key, deleted near the end.

Filed under: local-llm, llm, ai-agents, benchmarks

Sources#

Harness repository (private, local): agent-playground/bench/run-scenario.sh before and after commit 9e978a4 ("bench: apply harness-isolation patch to run-scenario.sh", 2026-10-15). The redirect, counter, peek scan, META definition and steer_prepare lines are quoted from it. Plan notes recording the zero-tool rows: commits 0ebca99 and 5112f67.
Benchmark result rows: results-tdd-tddlong13.jsonl, results-tdd-tdd64long13.jsonl, results-steerrevlong11-steerreview.jsonl and every other results*.jsonl for the zero-tool count.
OpenCode session database (opencode.db, opened read-only): tool-call commands, call order, timestamps and tool-part counts. Only tool-call commands are quoted.
Compaction and token audit of the same runs (2026-10-12): the 61 reasoning mentions and the lost-task compaction, re-derived from the database for this post.