My isolation guard checked itself

· 9 min read
Contents

Follows The mutant was killed after it switched a real GPU, which built the isolation this post tests.

I used to think a guard test fails when the thing it guards is gone. That is what the word means. Mine guarded the wall between my Go CLI's tests and my live LAN proxy, which starts a real model server on a real GPU box whenever a request asks for a model that isn't running. I wrote the guard, described it in a post, and trusted it the way you would trust an examiner who lets the student bring the answer key: nothing ever comes back wrong. I took that as a good sign. So, I imagine, would the student.

The wall exists because of the incident in that last post. A mutant, a deliberately broken copy of the code run against the tests to see whether any test notices, sent a real model load through the live proxy. The fix gave the CLI's tests a fake config that nothing answers, plus a guard test to prove it. That post described the guard in one sentence: it fails if the default config path is outside the test's temp dir. The sentence was true. Then a review, which in this project writes its own mutants for every change, removed the line that points the tests at that temp dir, and on the right shell the guard passed anyway. The examiner, it turned out, had been copying from the student.

The isolation layer and its guard#

The CLI is part of that proxy's Go program: the commands that talk to it. Its tests run behind an isolation layer. A TestMain writes a test config with every address moved to 127.0.0.1:9, the discard port, and every ssh target to nobody@lrp-test.invalid. It also points XDG_CONFIG_HOME at a temp dir, so a test that falls back to the default config path finds nothing there.

The guard's first check looks at that last part:

func TestTestConfigReachesNothing(t *testing.T) {
	if p := defaultConfigPath(); filepath.Dir(filepath.Dir(p)) != os.Getenv("XDG_CONFIG_HOME") {
		t.Fatalf("default config %s is outside the test's XDG_CONFIG_HOME", p)
	}

Read it as a question: is the default config path two directories below XDG_CONFIG_HOME? The default path is built from os.UserConfigDir(), which on Linux returns $XDG_CONFIG_HOME when it is set and non-empty, else $HOME/.config. So the guard asks the environment variable where the config lives, then checks the answer against the same environment variable. It grades the exam against the answer key the student handed in with it. Full marks, every time the key and the exam agree, which is every time the student wrote both.

The mutant that needed the right shell#

The review's mutant was one line: drop the os.Setenv("XDG_CONFIG_HOME", tempdir) from TestMain. Without it, the tests run with whatever the shell had.

From a shell with XDG_CONFIG_HOME=~/.config set, the default path becomes the developer's real hosts.json, which on that machine exists. That is the file with the live addresses, the exact thing the isolation layer exists to keep out of reach. The guard computes the path from XDG_CONFIG_HOME, compares it with XDG_CONFIG_HOME, finds them equal, and passes. Mutant SURVIVED.

From a shell without the variable, the same mutant was KILLED. os.UserConfigDir() falls back to $HOME/.config, os.Getenv returns an empty string, the two sides differ, and the guard fails as designed.

One mutant, two verdicts, depending on a line in someone's shell profile. A test whose verdict depends on a dotfile is a horoscope with a go.mod. A mutation run on a machine without that variable reports this guard as working. The gap only shows where the variable is set, and a run anywhere else gives no hint that it exists.

Nothing reached the network in that run. Every mutant run now goes through unshare -rn, the fix from last time, so there was no route off the box. That fix held. It was never a fix for guards. The point of the guard was to fail before it came to that. Belt and braces, and the braces had been painted on. Had the Setenv line ever gone missing for real, a plain go test outside any namespace would have read the real config, on a shell like that one, with the guard waving it through.

Fix one: compare against what TestMain chose#

The repair is small. TestMain keeps the temp dir it picked in a package variable, testXDG, and the guard compares against that value instead of reading the environment back. Now the expected value comes from the decision, and the actual value comes from the code under test. If the Setenv goes, defaultConfigPath() follows the shell, testXDG still names the temp dir, and the guard fails on every machine.

A guard is only as independent as the place it gets its expected value from. Mine got it from the thing it was guarding. I had written both the student and the examiner, and then sat them at the same desk.

The second gap: a test can name the example config again#

The review didn't stop at the environment. The test config replaces every address, but nothing stopped a new test from naming the repo's example config directly, the way the test in last time's incident did. The example config is a copy of the real one, so its addresses are the live ones. In hindsight, the word "example" was doing a lot of unsupervised work in that file name.

Three mutants that pointed a test back at the example config survived. One of them, combined with last time's mutant that ignored -proxy, is exactly the original path to the live proxy: the same file, the same fallback, the same listen address. The test config protects the tests that use it. It had no opinion about a test that doesn't.

Fix two: scan the sources, not only the values#

Values can be checked at run time. Intentions have to be checked in the source. Two tests cover the two sides, next to the guard:

  • A source scan. It reads the package's other *_test.go files and fails when one of them names hosts.example.json, a LAN address prefix, a .local name or /.config/.
  • A check of the generated config. It goes through the config that TestMain wrote: every user@host must end in .invalid, and every IPv4 literal must be 127.0.0.1.

The first protects against the next test I write. I have met the man who will write it, and I don't trust him. The second protects against the next field someone adds to the config and forgets to rewrite.

The checks that were too narrow#

Those were the first versions, and the second review pass found both of them too narrow. They leaned on regexes for IPv4 literals and for user@host, and the .local check was a substring match on .local". Three kinds of address walked straight past them: http://somehost:8080, http://[fd00::1]:8080 and http://name.local:8080. A bare name has no IPv4 shape. An IPv6 literal has no IPv4 shape either. And .local" only matches a name at the very end of a string, which name.local:8080 is not.

A toy version of the narrow check and the parsing check shows the difference. It is not the proxy's code, just the two ideas side by side:

// narrow: an IPv4 literal other than 127.0.0.1, or a name ending in .local".
func narrow(line string) bool {
	for _, ip := range ipv4.FindAllString(line, -1) {
		if ip != "127.0.0.1" {
			return true
		}
	}
	return strings.Contains(line, `.local"`)
}

// parsed: any http(s) literal whose host is not 127.0.0.1, "p" or empty,
// or .local followed by a word boundary anywhere.
func parsed(line string) bool {
	for _, lit := range urlLit.FindAllString(line, -1) {
		u, err := url.Parse(lit)
		if err != nil {
			return true
		}
		switch u.Hostname() {
		case "127.0.0.1", "p", "":
		default:
			return true
		}
	}
	return localWB.MatchString(line)
}

with ipv4 as \b(\d{1,3}\.){3}\d{1,3}\b, urlLit as https?://[^"\s]* and localWB as \.local\b. Run over a few lines, two of them harmless or already caught:

$ go run .
base := "http://127.0.0.1:9"       narrow=false parsed=false
base := "http://192.0.2.10:8080"   narrow=true  parsed=true
base := "http://somehost:8080"     narrow=false parsed=true
base := "http://[fd00::1]:8080"    narrow=false parsed=true
base := "http://name.local:8080"   narrow=false parsed=true
host := "name.local"               narrow=true  parsed=true

I wrote the toy's narrow check with the list of escapes open in front of me, and it still lets three of them through. url.Parse and Hostname() don't care what shape the host has; Hostname() even strips the brackets off the IPv6 literal. In Go's regexp syntax, \b is an ASCII word boundary, so \.local\b matches before the : as well as before the quote.

The real checks got the same ideas, each where it fits. The source scan now also fails on any http(s):// literal whose host is not 127.0.0.1, a placeholder host p, or empty, and it matches \.local\b. The config check now walks the JSON: every string containing :// must parse to host 127.0.0.1, next to the old user@host and IPv4 rules. Then each of the review's six escape cases, the three addresses above among them, went back in as a mutant, run against the scan test alone, so no other test could kill it on the scan's behalf. All six were killed.

What the guard checks now#

The guard now gets its expected value from TestMain, the source scan reads every test file I add to the package, and the config walk reads every string in the config TestMain writes. Last time's one-line description of the guard never said who decided where the temp dir was, and that turned out to be the part that mattered. Here is the line I should have written: the guard fails if the default config path is anywhere other than the directory TestMain made, whatever the shell says. The old sentence was shorter, it was true, and it passed every check I gave it. So did the guard. The examiner brings its own answer key now.

Filed under: go, testing, linux, local-llm

Sources#