The secret filter made my security test pass

· 9 min read
Contents

Builds on The model that only says RUN, which explains how a fake model drives the real agents in these tests.

My end-to-end suite rests on one belief: if the marker doesn't come back, the command didn't run. Every "blocked" verdict for six coding agents is that belief applied to a tool result. It is a very convenient belief. It holds right up to the moment something between the command and the test eats part of the output, and then a command that ran reads as a command that was stopped.

The rule I was adding was simple. The guard in my agent-config templates hard-denies destructive shell commands for all six (OpenCode V1 and V2, pi, omp, Hermes Agent and Claude Code), even when an agent runs with auto-approval, and it was getting one more job: deny a recursive search (grep -r, rg --hidden, ag -u) when a real .env file sits under the search root. Every test project in the suite carries a .env with SECRET=hunter2, a fixture value, there to be found. The new end-to-end case asks each agent to run grep -rn SECRET . and expects it to be blocked.

Then the standard check, the one I never skip: run the new case against the old guard, which has no such rule. It must fail there. If it passes on code that can't possibly block it, the test is measuring something else.

Five agents failed, as they should. The grep ran. Hermes Agent passed.

How the suite tells "ran" from "blocked"#

The driver doesn't ask an agent what happened. It gives each case's command a tail, <command>; echo __EXECUTED__. A fake model asks the agent's shell tool to run that line, and the driver looks for __EXECUTED__ in the tool result that comes back. Marker present: the command ran. Marker missing: something stopped it, and for a must-block case that's a pass.

Read that last part again with a security hat on. Anything that removes text from a tool result turns into a block: a truncated result, a length cap on what the fake echoes back, a banner that pushes the marker out of view, a filter that rewrites output. None of those is a guard. All of them would score as one.

The control that passed on the old guard#

Hermes "blocked" a command on a guard that had no rule to block it with. So I stopped trusting the verdict and printed the raw tool result instead. This is the line the probe printed:

{"output": "./.env:1:SECRET=hunter...__", "exit_code": 0}

Exit code 0. The grep ran, found the fixture, and printed it. What it didn't print was __EXECUTED__, and it also didn't print the last character of the secret, the line break, or the end of the line. Something had been at this string.

Nothing in the old guard could block that grep, so the pass was a bug report, and it was about my test.

Where the marker went#

Hermes has a setting, security.redact_secrets, that masks credentials in the text it handles. It is on by default, and my template keeps it on, on purpose: a secret filter is exactly what you want around an agent that can read a .env. In Hermes at commit afe3cbd, the pin my templates use, the mask for KEY=value style assignments lives in agent/redact.py:

_SECRET_ENV_NAMES = r"(?:API_?KEY|KEY|TOKEN|SECRET|PASSWORD|PASSWD|PASS|PW|CREDENTIAL|AUTH)"
_ENV_ASSIGN_RE = re.compile(rf"([A-Z0-9_]{{0,50}}{_SECRET_ENV_NAMES}[A-Z0-9_]{{0,50}})\s*=\s*(['\"]?)(\S+)\2")

The value is (\S+): everything up to the next whitespace. The masking function's docstring explains the rest: "Mask a log token — 18-char floor, preserves 6 prefix / 4 suffix; empty → ***." A token shorter than 18 characters becomes ***; a longer one keeps its first six and last four characters.

Now apply that to the tool result as JSON text, where the line break is still the two characters \n, and not whitespace at all. I copied the pattern and the mask into a small script and fed it the tool result as Hermes would have built it before the mask ran. It is a reduction, not Hermes itself; each pair of lines is the input, then the masked text:

$ python3 -I mask_marker.py
{"output": "./.env:1:SECRET=hunter2\n__EXECUTED__", "exit_code": 0}
{"output": "./.env:1:SECRET=hunter...__", "exit_code": 0}
{"output": "./.env\n__EXECUTED__", "exit_code": 0}
{"output": "./.env\n__EXECUTED__", "exit_code": 0}

The second line is byte for byte what the probe printed from the real Hermes. \S+ started at hunter2, ran through the escaped line break, through __EXECUTED__, through the closing quote and the comma, and stopped at the space before "exit_code". That's 23 characters, enough to clear the floor, so the mask kept the first six, hunter, and the last four, __",. The marker went into the masked token with the secret.

There's a detail in there I like even less. hunter2 is seven characters long, and the mask printed six of them. On its own, a seven-character value is under the floor and would become ***. It only got the head-and-tail treatment because my marker rode along and made the token long enough. The redaction hid the tail, not the value, and the tail it hid was my test.

The second pair is the same check with grep -rln, which prints only file names. There is no SECRET= in the output, so there's nothing to mask, and the marker comes back untouched.

The fix passed too, which was worse#

Then I ran the case against the new guard, the one with the .env rule. Hermes passed again. Every other agent blocked the search, and Hermes passed, and for about as long as it took to remember the previous section, that looked like success.

It was the same eaten marker. A pass in Hermes now meant nothing in either direction, so the only question left was whether the rule had actually fired, and the raw output said the grep had run. The new rule did not work in Hermes at all. Two green results from two guards, the same eaten marker behind both, and behind the second one a rule that had never fired.

The hook was told the wrong directory#

The .env check needs one fact: the directory the command will run in. Every agent hands that to the guard's hook along with the command. The Python guard's unit tests passed, and so did a direct call of the hook, because both gave it the right directory.

So I logged the payload Hermes actually sent. Three directories turned up, all different:

  • cwd in the payload was Hermes' own install directory. Its Python process had changed directory.
  • $PWD in the hook's environment was the project, where the terminal ran.
  • $TERMINAL_CWD was a third directory, Hermes' configured terminal working directory.

The command ran in the project. The guard scanned Hermes' source tree for a .env, found none, and allowed the search. Hermes' hook code is plain about where the field comes from; agent/shell_hooks.py fills it in like this:

cwd = str(Path.cwd())

That is the working directory of the Hermes process, not of the terminal it runs commands in. The example payload in the hooks documentation shows "cwd": "/home/user/project", which is what I had read into the field.

Why had nothing else noticed? Because the guard's other path rules match by suffix: a path containing /.git/hooks/ or /.ssh/ is caught wherever it sits, so a wrong cwd costs them nothing. The .env check has to know where the command is standing. The suffix rules never needed to.

What changed#

Three fixes, each where its problem lived:

  • The test config, not the template. The driver turns redact_secrets off in the throwaway config it writes for each test run. The template I ship keeps it on, because the mask is a good idea everywhere except in a test that reads output for a living.
  • The case. The search case uses grep -rln, file names only, so there is nothing for any mask to eat.
  • The hook. For shell tools, the Hermes hook now checks $PWD and $TERMINAL_CWD as well as cwd, and denies the search if any of them has a .env under it.

Afterwards the search is blocked in all six agents, and on the old guard it runs in all six, which is the half I care about. The control has to be able to fail.

Then the uncomfortable recheck. If redaction could fake one block, it could have faked others, and Hermes had a whole suite of "blocked" verdicts for things like cat .env and env. With redaction off, the full Hermes suite passed 34 of 34, so those blocks were real.

Silence is not a verdict#

A test that reads a missing string as "blocked" will score anything that eats output as security: truncation, a length cap, a wrapper's banner, a secret filter doing its job. The one defence I know is the negative control. Run the case on the code that can't block it, and when it passes there, read the raw output before believing anything, including the next green result.

The secret filter is still on in every Hermes install my templates set up. It is off in exactly one place, the test config, and that is the only place where the thing it hides is the evidence.

Update, 2026-10-26: the suite now checks the side effect as well. Every case's project has a victim/ directory, and a shell case whose command removed it counts as executed even if the marker never came back. So rm -rf victim; exit 1, which fails before the marker is printed, can no longer read as blocked. A seven-check unit test pins the verdict logic; two of its checks fail on the old logic.

Filed under: ai-agents, security, testing

Sources#