My unit tests never typed `; echo`

· 7 min read
Contents

Builds on My guard blocked bash -c. Models write bash -lc., which explains the guard and its command-position rules.

Both guards agreed on more than 700 cases. Four coding agents then failed one end-to-end case, and the failures were right.

The guards are the TypeScript and Python halves of the guard in my agent-config templates: one shared file of regex rules that hard-denies destructive shell commands even when an agent runs with auto-approval. Both halves have their own unit tests, and the two must agree. The case that failed was git log -S 'DROP TABLE' --oneline, which searches history and destroys nothing. It has been in the end-to-end suite as a near miss, a harmless command that has to keep running, ever since a harness's own *DROP TABLE* permission glob blocked it. I had just made the SQL rule stricter, every unit case was green, and the agents now refused to run a search.

Four commands that weren't destructive#

The round started with false positives. Some rules matched their words anywhere in a line, not only where a command can start, and they denied ordinary work:

CommandDenied as
git commit -m 'Drop table legacy_users in V7 migration'destructive SQL
echo 'DROP TABLE IF EXISTS tmp_import;' > V7__drop_tmp.sqldestructive SQL
mvn -pl deploy testpublishing
grep -rn 'git push --force' docs/a force push

A commit message that describes a migration. A migration being written to a file. A Maven module that happens to be called deploy. Documentation being searched for the very command it warns about. The rules weren't wrong that those words are dangerous. They were wrong about every one of these lines. This post follows the SQL rule.

For the SQL rule, two exemptions went in for the first two rows: a line that is only git commit with plain or quoted arguments, and a line that is only an echo or printf redirected into a .sql file. While I was in there, I found the opposite problem.

The exemption that skipped the whole line#

The SQL rule starts with a negative lookahead that exempts readers. A line that searches or prints files can mention DROP TABLE all day; it is meant only to read. The exemption, as it stood:

^(?!\s*(?:grep|egrep|rg|ag|ack|git\s+(?:grep|log|show|diff)|sed\s+-n|awk|less|head|tail|cat)\b)

Look at what it checks: that the line doesn't start with a reader. Nothing about where the reader ends. So this line never reached the destructive-keyword part of the rule at all:

cat a.sql; psql -c 'DROP TABLE users'

Print a file, then drop a table. Neither guard looked twice, because the first word was respectable. That is the lesson of this hole: an exemption is a promise about the whole line, so it had better read the whole line.

The tightened version did. The whole line has to be the reader, optionally piped into other readers. A ; or & outside quotes ends the exemption, and so does a $ or a backtick anywhere outside single quotes:

^(?!\s*READER\b(?:'[^']*'|"[^"$`]*"|[^'"|;&$`\n])*(?:\|\s*(?:head|tail|less|more|wc|sort|uniq|cut|grep|rg)\b(?:'[^']*'|"[^"$`]*"|[^'"|;&$`\n])*)*$)

READER stands for the same alternation as before. Every unit case passed, both guards agreed on all of them, and I ran the end-to-end suite feeling quite pleased with myself.

Green in both guards, red in four agents#

The end-to-end suite doesn't trust a harness's summary of what happened. It drives each real agent binary with a fake model that asks for one shell command per case, and to see whether the command really ran, the driver sends it with a tail: <command>; echo __EXECUTED__. If __EXECUTED__ comes back in the tool result, the command ran.

So the agents never saw git log -S 'DROP TABLE' --oneline. They saw this:

git log -S 'DROP TABLE' --oneline; echo __EXECUTED__

The tightened exemption stopped at the first ;, the line fell through to the keyword part, and DROP TABLE was right there inside the quotes. OpenCode V2, pi, omp and Claude Code all refused the search. That was every harness that ran: I had stopped OpenCode V1 by hand before it finished, and Hermes never started in that run, because of a setup problem in the test environment (its virtualenv's Python lived in another directory).

The four red results were right. The rule did exactly what I had told it to, to a harmless command. Every unit case had the bare command. None had the command plus a tail, and a tail is the most ordinary thing in the world: the next command in a chain, a && echo ok, a ; echo done. My unit tests had tested the command I imagined, and the agents ran the command the driver actually sent.

The rule that shipped#

The final exemption accepts the reader's own segment and then lets the rest of the line run anything that names no SQL keyword:

^(?!\s*READER\b(?:'[^']*'|"[^"$`]*"|[^'"|;&$`\n])*(?:[;&|](?![^\n]*(?:(?:DROP|TRUNCATE|DELETE|FLUSHALL|FLUSHDB)\b|\.dropDatabase\b))[^\n]*)?$)

After a reader, a ;, & or | may follow, but only if nothing after it says DROP, TRUNCATE, DELETE, FLUSHALL, FLUSHDB or .dropDatabase. The cat …; psql line still names DROP after its ;, so it gets checked. The near miss names nothing after its ; but echo __EXECUTED__, so it's exempt.

To watch the three versions side by side without the rest of the rule file, I wrote exempt_demo.py: 27 lines that put the three lookaheads in front of a stand-in for the keyword part, .*\bDROP\s+TABLE\b, and ran four lines through them. It is a reduction, not the guard's real rule: fires means the stand-in rule would apply, exempt means the lookahead let the line through.

import json, re, sys

# The SQL rule's three leading lookaheads, with a stand-in for the keyword part
# that follows them. Not the guard's full rule.
READER = r"(?:grep|egrep|rg|ag|ack|git\s+(?:grep|log|show|diff)|sed\s+-n|awk|less|head|tail|cat)"
SEG = r"""(?:'[^']*'|"[^"$`]*"|[^'"|;&$`\n])*"""
ORIGINAL = rf"^(?!\s*{READER}\b)"
TIGHTENED = rf"^(?!\s*{READER}\b{SEG}(?:\|\s*(?:head|tail|less|more|wc|sort|uniq|cut|grep|rg)\b{SEG})*$)"
FINAL = (rf"^(?!\s*{READER}\b{SEG}"
         r"(?:[;&|](?![^\n]*(?:(?:DROP|TRUNCATE|DELETE|FLUSHALL|FLUSHDB)\b|\.dropDatabase\b))[^\n]*)?$)")
KEYWORD = r".*\bDROP\s+TABLE\b"  # stand-in

STEPS = {"original": ORIGINAL, "tightened": TIGHTENED, "final": FINAL}
LINES = [
    "cat a.sql; psql -c 'DROP TABLE users'",
    "git log -S 'DROP TABLE' --oneline",
    "git log -S 'DROP TABLE' --oneline; echo __EXECUTED__",
    "grep -rn 'DROP TABLE' migrations | wc -l",
]

if sys.argv[1:] == ["--json"]:
    print(json.dumps({"steps": {k: v + KEYWORD for k, v in STEPS.items()}, "lines": LINES}))
else:
    print("original   tightened  final      line")
    for line in LINES:
        cells = ["fires" if re.search(p + KEYWORD, line) else "exempt" for p in STEPS.values()]
        print("".join(f"{c:11}" for c in cells) + line)

$ python3 -I exempt_demo.py
original   tightened  final      line
exempt     fires      fires      cat a.sql; psql -c 'DROP TABLE users'
exempt     exempt     exempt     git log -S 'DROP TABLE' --oneline
exempt     fires      exempt     git log -S 'DROP TABLE' --oneline; echo __EXECUTED__
exempt     exempt     exempt     grep -rn 'DROP TABLE' migrations | wc -l

Each column tells one chapter. The original exempted everything that started with a reader, including the table drop. The tightened version caught the drop and also the near miss with its tail. The final one catches the drop and lets the tail through. Node 22.22.0 gives the same four rows from the same patterns. That matters, because the TypeScript guard runs them in a JavaScript regex engine and the Python guard in re (Python 3.14.8 here); Node's run covers V8's engine, not every JavaScript runtime.

The search itself is as harmless as it looks. git log -S is the pickaxe; in git 2.56.0's man page it will "Look for differences that change the number of occurrences of the specified (i.e. addition/deletion) in a file." It reads history. The only thing it changes is my opinion of my unit tests.

Test the shape the agent sends#

The fix to the rule swapped the piped-reader group for one optional tail that may hold anything without a SQL keyword. The lesson for the tests: an exemption's unit cases should include the shape the agent actually sends. That means the bare command, the command plus ; echo done, and one with a dangerous tail that must still be caught. An exemption that only ever sees the clean command has only been tested on the clean command.

After this round and the ones that followed, the unit tests count 833 cases in the Python guard and 870 in the TypeScript one, and the two guards agree. The end-to-end suites passed on all six harnesses: OpenCode V1 34/34, OpenCode V2 34/34, pi 31/31, omp 35/35, Hermes 34/34 and Claude Code 39/39.

Filed under: ai-agents, security, testing, bash

Sources#