My drift check said 0 differences. It checked 4 arrays.

2026-10-31 · 10 min read

Contents

My LAN proxy for local LLMs keeps its own catalog of how to serve each model, models.json. Each of the two GPU boxes behind it keeps the same settings again in its own models.sh, a bash script. I wrote a drift check to compare the two, and its first run said "0 differences".

I read that as a sentence about the config files. It was a sentence about my parser, which had pulled four arrays out of two bash scripts of about 500 and 600 lines, compared them, and reported back with total calm.

I should have known better. A new checker that comes back with zero on its first run has told me exactly one reliable thing: that it ran. Everything else in the zero is a claim about what it read, and on day one nobody has checked that claim, least of all the checker.

Two copies of the same settings#

The proxy is a small Go service in front of the two GPU boxes, an RTX 3090 desktop and an AMD Strix Halo mini PC, which starts and stops llama.cpp servers on them as requests arrive. The per-model serving settings exist twice:

the proxy's catalog, models.json: the model file, the context length and the llama.cpp preset options (the server's command-line flags as named options, lazy-mode for --lazy-mode) for each model key, the short name a model goes by, such as laguna-xs;
each box's own models.sh, which the benchmark scripts on that box source. It belongs to a separate benchmark project, maintained on its own.

The catalog was imported from the boxes' rendered presets on 24 October and matched them option for option that day: 31 of 31 presets on the 3090, 24 of 24 on the Strix Halo. Since then each side has been retuned on its own. A retune on one side doesn't reach the other, and nothing noticed. My own design doc even said the proxy "never holds a second copy of the roster". It had held one since the import, and the doc went on denying it in the present tense.

So the backlog got an item: engine roster <host>, the proxy command that lists which catalog keys a box offers and why it skips the others, should also read the host's models.sh and print the differences.

models.sh is a program, not a file of values#

models.sh is bash. A few of its lines:

GGUF[laguna-xs]="Laguna-XS-2.1-Q4_K_M.gguf"; CTX[laguna-xs]=65536; ROLE[laguna-xs]="..."
SAMPLER[laguna-xs]="--temp 1.0 --top-p 1.0 --top-k 20"
SAMPLER[qwen38-27b-192k]+=" --ubatch-size 256"

Associative arrays per model key, several statements to a line, sampler flags as one string, appends. Around them:

a helper, context_profile <key> <base> <ctx> "<note>", that derives about twenty longer-context variants by copying the base key's arrays;
loops over literal key lists that append more flags;
a loop over every key that appends --n-predict 32768 to every sampler string;
unset 'REASONING[...]', a per-host case on the hostname, && guards;
on the Strix Halo, a SERVE_ARGS[key] array of load-bearing server flags such as --lazy-mode on, --rope-scaling yarn and -b 2048 -ub 2048.

The 3090's script is about 53 KB, 504 lines and 73 keys; the Strix Halo's about 65 KB, 587 lines and 93 keys.

Read it, never run it#

The exact way to get the values is to run the script: source it in bash and dump the arrays. That's how the benchmark project's own config renderer does it. But it's another project's code, on another machine, and the proxy's host-side agent, the same Go binary run over ssh with one JSON request, never runs scripts by design. I wasn't going to make an exception for a file I don't maintain. So the checker parses only what it can read safely, and says what it didn't compare.

The agent got one new read-only operation:

read one regular file named models.sh under $HOME, at most 1 MiB;
open it through Go's os.Root, so a link can't lead outside $HOME, not even one swapped in between the check and the open;
open it with O_NONBLOCK, so a FIFO named models.sh returns an error at once instead of hanging the agent.

The existing file-read operation stayed restricted to the agent's own unit files.

Version 1, and the zero#

The first parser read lines: NAME[key]="value", context_profile calls (positional, so readable), unset, and loops over literal keys, whose fields it marked "computed". A field touched by += or a $ expansion was marked computed too. It compared four things per key: file name, context length, chat template and sampler flags.

On read-only copies of the real files it found 0 differences in 34 shared keys on the 3090 and 48 on the Strix Halo. Two zeros, one per box. I keep a list of things that report success while on fire, and it has a whole page for round numbers on a first run. I didn't open it.

Then the usual test of a checker: plant faults in a copy and see whether it finds them. A context length, a temperature, a GGUF model file name, a chat template. All four found, plus the variants that inherit them through context_profile, six lines in all. Four plants, four catches, a bonus for the inherited ones. That is precisely the result that makes me stop looking.

The review that kept looking#

A review of the checker came back NEEDS WORK with three medium findings.

The zero claimed more than it checked. The compare covered four arrays, and only the host's side of the flags. Not compared: the Strix Halo's SERVE_ARGS, the MTP (multi-token prediction) settings, mmproj (the multimodal projector file), the MoE expert offload, and any option the proxy sets that the host doesn't.

The parser could invent differences and hide others. Its own doc comment said "a difference is never invented"; the comment was the first thing in the file to invent something. The review came with 17 small bash shapes that broke it, aimed at a parser that had looked me in the eye and promised, in writing, never to invent a difference:

applied as if unconditional: an assignment inside if ... fi;
ignored: a call to another function;
first value used for --temp 0.6 --temp 1.0, where llama.cpp takes the last;
not recognized: unset "X[k]" with double quotes, a loop index written ["$k"], do on the next line;
a loop body ended early by the word "done" inside the loop's first line;
silently dropped: [ c ] && CTX[x]=1, one-line case arms, X=([k]=v).

The real scripts used none of these yet. "Yet" is doing real work in that sentence: the 3090's script already had a per-host case and && guards.

The useful output was buried. On the 3090, 25 keys that exist only on the other box showed up as "only in the catalog".

My planted faults had tested the shapes I knew the files used. The review's shapes tested the ones they might use next, and that list was the more useful of the two.

Version 2: what it can't follow, it doesn't compare#

The second parser keeps a stack of blocks: if, case, while, until, for, function bodies and { ...; } groups. On top of that:

an assignment inside a block or after a guard is marked computed;
a loop over every key that replaces a value marks the whole array computed;
a loop that only appends (the new value starts with the old one) is noted, with the flags it adds;
a call to another function marks what that function assigns for the literal keys it's passed;
source marks everything computed;
here-docs are skipped only when their delimiter line is found;
line continuations are joined.

A computed field is not compared. It never turns into a value.

The compare now runs both ways. Long flags map to preset option names (--lazy-mode becomes lazy-mode), plus a few short ones (-ctk, -b); the last repeated flag wins; MTP_N with MTP_DEFAULT, MMPROJ and NCMOE map to their own options. Options the proxy sets that the host's exact flags lack are reported, except flags a loop over every key appends. And only keys the host actually offers are compared, which took care of the 25 strangers.

The review's 17 shapes became test cases. Mutation checks ran in a network-less namespace, the way every mutant on this proxy runs now, over three rounds. By the end every mutant was killed, though not all on the first try. Two needed a stronger test first: at first one was caught only by the 120-second package timeout, and the other only by a case that couldn't tell it from the original code. One mutant survived because the branch it changed was redundant; the branch was deleted, and the mutant with it.

A second review pass gave PASS with fixes, plus one new medium. A { ...; } group inside an if closed the if early, so a later conditional context line read as unconditional: a false difference again, from the parser whose whole job was now never to invent one. Two versions, one promise, broken in both. Consistent, at least. Fixed the same way, with groups pushing a block of their own, along with closers followed by a redirect (} > log) and printf -v into an array.

The first two-way run: 18 differences, all mine#

The first run of the two-way compare found 18 differences on the 3090, and every one of them was the same line: n-predict: host none, proxy "32768".

When a checker reports one thing eighteen times, the checker is my first suspect. The every-key loop writes its flag right after a closing brace, "${SAMPLER[$_k]:+${SAMPLER[$_k]} }--n-predict 32768", and the pattern that collects appended flags required whitespace before --. The flag was there; my regex couldn't see it. After the fix: 0 on the 3090.

The real one: an output cap one box never got#

The same line on the Strix Halo was real.

How many times depends on which run you ask. The offline run over read-only copies showed it 21 times, because that run compared every catalog key of that engine, model files present or not. The shipped check compares only the keys the box actually offers, files on disk; live on the box on 31 October, it found 7 differences, all this line. The 3090, live, came back with 0 differences and one catalog-only key, a proxy-only variant added the day before.

The line matters. The proxy caps output at 32768 tokens for requests that send no max_tokens. That cap went in on 27 October, after a thinking request without a limit held the single server slot for 592 seconds, 65k tokens of reasoning. The 3090's script got the same cap the same day. The Strix Halo's script never did: it has no n-predict anywhere, which a separate read of the file confirmed.

So the drift check's first real finding is a guard that the benchmark side of one box is missing. That's the benchmark project's call, not mine, and it has put the call off on purpose: the item is logged to decide after its current benchmark round, because changing the cap mid-round would split the comparison arms. The check's job was to make the gap visible. It did that, after two parsers, two reviews and one bug in my own regex.

What a zero is worth now#

On the 3090 the check says 0 again, the same number as on its first day. I read it the way I now read every zero: as the opening line of an argument, not the end of one. This time a field it can't follow counts as not compared instead of as a value, the compare runs both ways, and the checker has already been wrong in ways I could see and fix. I trust it more for that. Not completely. The regex has my handwriting on it.

Filed under: go, bash, local-llm, testing

Sources#

os.Root (Go 1.24+): methods only reach files beneath the root, and symbolic links may not point outside it.
Traversal-resistant file APIs: the Go blog's introduction to os.Root.
unshare(1): -n/--net, the network-less namespace the mutants ran in.