The mutant was killed after it switched a real GPU

· 10 min read
Contents

I test my fixes by breaking them on purpose. The method is mutation testing, done by hand: copy the module, undo one condition (that copy is a "mutant"), run the copy's tests, and record KILLED if a test fails or SURVIVED if none does. KILLED is the result you want; it means the tests noticed. The code in question is a small Go program on my LAN that sits in front of the GPU boxes serving local language models and decides which model server runs where. I like the method because its verdict is one word. It had never occurred to me to ask what a mutant gets up to between being created and being killed.

The second mutant run for one feature came back 34 mutants, 33 killed. Mutant U19, to use its label in that list, was one of the 33, and its verdict was correct: I had broken one condition, and a test failed. Both runs of the list also loaded a model on one of the real GPU boxes, the second time a minute after someone else's multi-day queue of benchmark jobs had started on that box. The benchmark then waited just under half an hour for a model server nobody was using. The report has a column for whether a test failed. It has no column for what the broken code did on its way to failing, and I would now like that column very much.

The CLI and its one fallback#

The program is two things in one Go binary. It is a LAN proxy in front of GPU boxes that serve local LLMs through llama.cpp routers, and it is a CLI that talks to that running proxy: model load <name>, model status, generators for client configs. The proxy owns the model servers on the boxes. A request for a model whose server isn't running makes it start that server, an engine switch, and after 15 minutes without requests it stops the server again so a benchmark can have the GPU. One GPU owner per box; the proxy never fights a benchmark. The rule is mine, and I'd like the credit now, before the rest of this post takes it back.

The CLI takes -proxy URL. Without it, it reads the proxy's config file and uses the config's listen address. That afternoon's feature let the CLI work with -proxy alone, no config file needed, and the new helper is short:

func proxyBase(configPath, proxyURL string) (base string, cfg *config.Config, err error) {
	if proxyURL == "" {
		if cfg, err = config.Load(configPath); err != nil {
			return "", nil, err
		}
		proxyURL = "http://" + cfg.Listen
	}
	return strings.TrimRight(proxyURL, "/"), cfg, nil
}

Mutant U19 and the test it was meant to fail#

Each mutant in this project is written by hand, one undone condition per copy. U19 was the obvious mutant for this helper, the one that says "-proxy still reads the config":

-	if proxyURL == "" {
+	if true {

The test it ran against points the CLI at a fake proxy and passes both flags:

srv := httptest.NewServer(f)          // a fake proxy on 127.0.0.1
cfg := "../../hosts.example.json"     // the repo's example config
modelMain([]string{"load", "<real model name>", "-config", cfg, "-proxy", srv.URL})

Under the mutant, -proxy is ignored and proxyBase takes the fallback: load the example config, use its listen. The example config is a copy of the real one. Its listen is the address of the live proxy, the one running as a service on that same box. So the test's model load went to the live service, the fake never saw a request, the test failed, and U19 went into the report as KILLED. Every word of that report line is true. It is a death certificate with the cause of death filled in correctly and nothing about the deceased's last errand, which was to ask the live proxy for a real model on a real GPU.

What the live proxy saw at 16:53#

The live proxy's journal saw the other half. At 16:53 it logged a POST /models/load for a real model, sent from the proxy box itself with the user agent Go-http-client/1.1. It switched a GPU box to the server for that model, which was up 1.3 s later. Then the client polled GET /v1/models/<name> every 2 s, which is exactly the loop the CLI's model load runs while it waits for a model to come up.

At 17:02 the mutant list ran a second time and the pattern came back, this time with three GET /clients/*.json requests carrying a-k, a model name that exists only in the tests.

Why the benchmark waited until 17:29:43#

At 17:01, between those two runs, the benchmark queue started on that GPU box. Its guard waits while any model server runs, which is the right thing for a guard to do and the reason the proxy stops idle servers at all. The server my test had started sat there idle at 300 MiB. So two careful designs, one of them mine, held each other in place: the guard waited for the server to stop, and the server waited out its idle minutes, as designed, both of them. Its idle clock was reset at 17:02 by the second run and again at 17:14 by my deploy of the proxy, since a restarted proxy starts its idle clocks over, so the 15 minutes ran from 17:14. The proxy's own idle stop came at 17:29:43, and the benchmark got its GPU.

Laid end to end:

  • 16:53, first mutant run: model load through the live proxy, server up after 1.3 s
  • 17:01, the benchmark queue starts and its guard waits
  • 17:02, second mutant run: the same load, plus the a-k config requests
  • 17:14, my deploy restarts the idle clock
  • 17:29:43, idle stop; the benchmark starts its work

One swapped condition, run twice, plus my own deploy, cost someone else's benchmark just under half an hour. Nothing in the mutant report changed. Reports don't do guilt; that part, it turns out, is my department.

How the request log named the client#

The benchmark's owner noticed first. They saw a model load from a client on the proxy box and asked me to find out what that client was. The journal gave it away. A load from the proxy box itself, a Go user agent, then a poll every 2 s: that is my CLI's model load, step for step. The test-only name a-k a few lines further down settled whose tests had been running. The request line logs the method, path, user agent and timing, and that was enough to name the client. My own tooling, the one whose job is to judge the tests, never noticed anything.

Two harmless mistakes and the path only a mutant takes#

Neither mistake was new, and each was harmless on its own for months:

  • A fixture that names a production address. The example config was copied from the real one, so the test tree contained the live proxy's address. Example configs copied from production are production addresses that happen to live in a test directory.
  • A test whose subject can act on the network. model load is a CLI command that sends a real request and then waits for a real model. That is its job.

The real code never brought them together. With -proxy set, proxyBase never reads the config, so the test never looked at listen. A mutant exists to run the code paths the real code doesn't take, and the fallback "no flag, read the config" is the first one it takes. Mutation testing runs deliberately broken code. It deserves the sandbox you would give any other untrusted code, and I had given it my LAN. I build broken programs on purpose and then let them out unsupervised; put that way, the surprising part is that nothing got out sooner.

The same class has a quieter member. A test that falls back to the default config path, through os.UserConfigDir(), reads the developer's real config. On Linux that is $XDG_CONFIG_HOME, or $HOME/.config when the variable is unset or empty, so on my machine it is the live proxy's own config.

Fix one: a test config that nothing answers#

A TestMain for the CLI package now writes testConfig: the example config with listen, every host's base_url and every host's ssh moved to addresses nothing answers on, with the model catalog files linked in beside it so the config still loads. It also sets XDG_CONFIG_HOME to a temp dir, so the default path finds nothing. The rewrite at its core:

d["listen"] = "127.0.0.1:9" // discard port: refused
for _, h := range d["hosts"].([]any) {
	h := h.(map[string]any)
	h["base_url"] = "http://127.0.0.1:9"
	h["ssh"] = "nobody@lrp-test.invalid"
}

Port 9 is the discard port from RFC 863. This box doesn't run a discard server, so a connection to it is refused at once instead of hanging. .invalid is the top-level domain RFC 2606 reserves for names that are "sure to be invalid", so no DNS server will resolve it. Every test in the package uses testConfig, and a guard test fails if the default config path is outside the temp dir or if any address in the test config is not one of those two.

Fix two: no route off the box#

The test config covers the addresses I know about. The mutant runs also go into a network namespace without a route off the box:

unshare -rn sh -c 'ip link set lo up && go test -count=1 ./cmd/...'

-n gives the command a new network namespace, and -r maps my user to root inside a new user namespace, which is what lets an unprivileged user configure the namespace's interfaces. No sudo. The ip link set lo up matters: unlike bwrap's --unshare-net, which brings loopback up itself, unshare leaves it down. A small demo module with three tests, one httptest round trip and two dials, shows both sides:

$ unshare -rn sh -c 'ip link set lo up && go test -count=1 -v .'
=== RUN   TestLoopbackWorks
    demo_test.go:19: httptest http://127.0.0.1:46069: 200 OK
--- PASS: TestLoopbackWorks (0.00s)
=== RUN   TestOffBox
    demo_test.go:24: off-box dial: dial tcp 192.0.2.10:8080: connect: network is unreachable
--- PASS: TestOffBox (0.00s)
=== RUN   TestDiscardPort
    demo_test.go:29: discard port: dial tcp 127.0.0.1:9: connect: connection refused
--- PASS: TestDiscardPort (0.00s)
PASS
ok  	demo	0.004s

Without the ip link set lo up, the httptest round trip fails with the same "network is unreachable" as the off-box dial, and every fake-server test in the suite would fail with it.

Re-run this way, U19 is still KILLED. The test now fails with dial tcp 127.0.0.1:9: connect: connection refused, nothing leaves the box, and the live proxy's journal shows no request during the run. I checked the journal, because the report would never have told me.

A network namespace closes the network and only the network. Files stay writable and a Unix socket at a path still connects, which is most of what the test-rig post was about. Here the damage went over TCP to the live proxy, and the same config names an ssh target for every host; the test config takes away the addresses, and the namespace takes away the route.

Rule eight#

I keep numbered rules for test rigs. Rule seven says the rig assumes the thing it tests is already broken. A mutant is broken by construction, so I can't even claim it caught me by surprise. Rule eight is new as of this afternoon: every mutant run, mine or anyone else's, goes through unshare -rn. Per-test discipline would have needed someone to notice that the example config carried a live address. A namespace for the whole run needs nobody to notice an address, and nobody, on the evidence of this afternoon, is exactly who was noticing addresses.

Filed under: go, testing, linux, local-llm

Sources#