Tests that could not fail

· 5 min read
Contents

Builds on The secret filter made my security test pass, where the same rule caught a test that couldn't fail.

One rule in my agent-config project has paid for itself several times: a new end-to-end case must fail on the old code. A test that passes before the fix and after it can't tell me whether the fix works.

The suite drives six real coding agents (OpenCode V1 and V2, pi, omp, Hermes Agent, Claude Code) with a fake model and checks whether each command ran. For a fix to the guard, the shared regex rules that block destructive commands, "the old code" usually means the old rule file. For the guard cases below I broke the guard on purpose instead: a copy whose "ask" verdict returns "allow". Every case that counts on that ask to stop a command should then fail. Two of the new ones didn't, and a third test, from a different fix, passed on the old code just as happily.

Case one: a deny list got there first#

My suite has a write mode that runs Hermes without --yolo. In -q mode Hermes refuses anything that asks for approval, because there is nobody to approve it. The new case was chmod 600 .env: the guard asks about it, so in this mode the command must not run.

With the broken guard, whose ask now meant allow, the command was still refused. Hermes has its own deny list in its config, separate from my guard, and that list holds *.env. The case never reached the guard's ask at all. It passed on Hermes' deny list and would have kept passing with my guard deleted. A security test that stays green with the security removed is the calmest kind of lie.

The replacement is ln -s victim link, which the guard also asks about and which Hermes' list doesn't mention. With the real guard it is refused. With the broken guard it runs. Only then did the case test the thing in its name.

Case two: the harness refused on its own#

The second Hermes case was execute_code, its tool for running code, again without --yolo. Again the broken guard changed nothing. Hermes in -q refuses execute_code by itself, so the case couldn't tell my guard's ask from no guard at all, and I dropped it. The guard's ask on that tool is covered by unit cases, which call the guard directly and don't need Hermes to get out of the way.

Case three: the variable came too late#

The third case belonged to a different fix: discovery that sent model-list requests to the proxy. The test set HTTP_PROXY to a dead port inside the running test, called discovery, and expected the fake model server to be found anyway. It passed on the new code. It also passed on the old code, which in real use went straight to the proxy and found nothing.

Bun and Node had read the proxy variables when the process started (Node only uses them at all with NODE_USE_ENV_PROXY=1, as in the run below). Setting one later changed nothing for the requests that followed. Rerun on a fake server at 127.0.0.2 and a proxy on port 9, where nothing listens:

// set the proxy variable after the process has started, then fetch
process.env.HTTP_PROXY = "http://127.0.0.1:9";
try {
  const r = await fetch("http://127.0.0.2:18431/v1/models");
  console.log("late HTTP_PROXY ->", (await r.json()).data.length, "models");
} catch (e) { console.log("late HTTP_PROXY ->", e.code ?? e.cause?.code ?? e.name); }

$ bun late_env.mjs
late HTTP_PROXY -> 2 models
$ NODE_USE_ENV_PROXY=1 node --no-warnings late_env.mjs
late HTTP_PROXY -> 2 models

That's Bun 1.3.10 and Node 22.22.0. The proxy is dead and both runtimes found the models, because neither asked the proxy. The test now starts discovery in a child process with the variable already in its environment, and on the old code it fails in both runtimes, which is the whole point of it. One version note: Bun 1.4.4, released on 2026-10-22, reads the variable on each request, so under it the in-process test would have worked in Bun, though still not in Node. The child process works in both.

The method confirms good cases too#

Breaking things on purpose is not only a way to find bad tests. A patch-tool case, patch against .git/config, did fail as it should once the git-internals rule was removed from the guard. That one was testing what it claimed. The method costs a guard variant with one decision flipped or one rule cut, and it answers the question a green run never answers: would this test notice if the thing it guards went away?

Only case three was a mistake in how the test was built. The other two were another layer saying no before my guard could, and catching that matters just as much: a test that passes because a different layer said no will keep passing the day my layer stops working. It will pass that day too. Passing is the one thing it was ever reliably good at.

Filed under: ai-agents, security, testing

Sources#

  • Bun 1.3.10 and 1.4.4, Node.js 22.22.0: local runs of late_env.mjs, 2026-10-26.
  • npm registry: bun: release date of 1.4.4 (2026-10-22).