The config said 32k. The engine still ran 64k.

· 9 min read
Contents

TL;DR — My Go proxy for local LLMs now reloads three settings on SIGHUP and refuses any other change by name. A reload changes what the proxy tells clients at once, but a running model server keeps the preset it was started with, so the two can disagree until the server restarts; the reload log line says so. Overlapping model-list refreshes are numbered, and only a successful read moves the number forward.

Changing one line of my LLM proxy's JSON config used to mean restarting the one process every client on my LAN talks to. The proxy is a small Go program: clients get one URL and one merged list of models from it, and behind that URL it starts and stops the model servers on a couple of GPU boxes as requests need them. The config says which models exist and under which names, so it is the file I edit, and a restart is a lot of ceremony for one line. So I taught the proxy to re-read its config on SIGHUP, the signal a daemon traditionally takes as "read your config again".

systemctl reload is a polite command, and the first entry on my list of things that report success on someone else's behalf. It runs the reload line, waits for kill to return and reports success, the way a courier reports a parcel delivered once it has cleared the fence. Its own man page is frank about how little that proves. Right under the example line ExecReload=kill -HUP $MAINPID, systemd.service(5) notes that reloading a daemon "by enqueuing a signal without completion notification" is asynchronous. The same paragraph recommends Type=notify-reload, a service type where the daemon reports when its reload is done; my unit uses the plain kill line. So the command that triggers my reload was never going to tell me whether it worked, and I don't take reassurance from a command whose idea of done is that the signal left. The journal would have to tell me.

What a SIGHUP reloads#

The program is the same proxy as in the mutant post. The GPU boxes serve their models through llama.cpp routers, and I call each model server an engine. The proxy runs as a systemd --user unit, so the reload is systemctl --user reload, wired as ExecReload=/bin/kill -HUP $MAINPID. On SIGHUP the proxy re-reads its model catalog, its aliases and its naming setting, without a restart.

Those three and nothing else. A ReloadFrom method walks the Config struct by reflection, compares the running config with the new file, and refuses any other difference by name:

listen changed: restart to apply

A file that fails to load or to validate keeps the running config. Either way the journal says which it was, reload: applied ... or reload: kept the running config: <why>, and systemctl reload reports success both times. I've stopped taking that personally.

The whitelist is the part I'd defend hardest. Every reloadable setting has to argue its way in with a reason it is safe to swap under running requests, and I am a hostile audience. A setting without one stays out, and a refusal that names the field beats a reload that quietly says "applied".

An unhandled SIGHUP is an exit#

Go's default for SIGHUP is not "ignore". The os/signal docs say "A SIGHUP, SIGINT, or SIGTERM signal causes the program to exit", unless signal.Notify has taken it over. The first version registered its handler after loading the config, so a reload that arrived during startup would have killed the proxy instead of reloading it. That gave my reload a second mode: during startup, it was a very effective off switch. A review caught it, and the handler now goes in before the config is read.

A small program shows both orders. It sleeps two seconds to "load its config", and with the argument notify it calls signal.Notify for SIGHUP first. The script, run.sh, sends kill -HUP half a second in:

./sighupdemo & pid=$!; sleep 0.5; kill -HUP $pid; wait $pid; echo "exit status $?"
./sighupdemo notify & pid=$!; sleep 0.5; kill -HUP $pid; wait $pid; echo "exit status $?"

and prints:

loading config...
./run.sh: line 3: 1401218 Hangup                     ./sighupdemo
exit status 129
loading config...
SIGHUP: reload
serving
exit status 0

The first run has no handler and dies as a hangup, exit status 129. The second registers signal.Notify first, prints the reload and keeps going.

Readers behind atomics and a lock#

A reload swaps values while requests read them, so every runtime reader moved behind an atomic pointer or a lock:

  • an atomic.Pointer[config.Config] in main, read by the closures that need the config;
  • another one in the engine manager, where each operation loads one snapshot at its start, so a reload in the middle of an engine start cannot mix two catalogs;
  • the alias map and the naming setting in the proxy, behind an RWMutex.

Proving under -race that route()'s alias read takes that lock took two tries, and the first try hid the race it was meant to show.

32k in the model list, 64k in the engine#

The design had a mismatch built in, and a review pointed it out while the reload was still being written. The proxy's model list reports each model's context length from the config, which overrides what the running server says about itself. Engines, though, get their settings only when they start, and keep that preset until the next start.

So take a reload that lowers a model's context from 64k to 32k. Clients are told 32k at once, while the engine serving that model still runs with its 64k preset. The other direction is worse. Raise the number, and clients are told about context the engine doesn't have, and a long prompt overflows. My proxy would be making promises on the engine's behalf, which is the exact habit I hold against systemctl reload, now in my own code.

There were three ways out: restart engines on every reload, refuse context changes while an engine runs, or say it out loud. I chose the last. A mismatch I have written down is at least one I know about. It is documented, and the reload log line now carries it:

a running engine keeps its old preset (ctx, options) until it next starts

A hot reload has two clocks. One is what the proxy says now. The other is what each long-running child process was started with. A reload moves the first and can't move the second, and the honest thing a log line can do is admit there are two.

The refresh that finished last#

The second problem was a race between two slow reads rather than two memory accesses. A review rated it MEDIUM.

After a reload, the proxy re-reads each host's model list, a slow call over ssh; the reload gives each host read 30 s. Separately, a periodic refresh re-reads the same lists every 10 minutes. If that periodic refresh had started before the reload, on the old catalog, and finished after the reload's own refresh, it overwrote the new result. The old list then stood for up to 10 minutes, while the journal said, with a straight face, reload: applied. That line went on the list too, as entry two, and I wrote it.

The review proved it with a test that holds the old read open, runs the new one to completion, then releases the old one. The old one won, by the simple method of finishing last.

Numbered refreshes, and only successes count#

The fix numbers the refresh calls. Each call takes a generation under the catalog lock when it starts. Per host, a result from an older generation than the one stored last is dropped.

A second review pass found the trade-off in that rule. Suppose the newer read fails. If its generation is stored anyway, the older read, which may well have succeeded, now looks stale and gets dropped too. A good result gets thrown away for a read that brought nothing. So the final rule: the stored generation advances on a success only.

c.mu.Lock()
c.rgen++
gen := c.rgen
c.mu.Unlock()
// ... per host, after the slow read, under c.mu:
if gen < c.rseen[i] {
	return // an older read that finished late
}
// error handling: record the error, return (no generation stored)
c.rseen[i] = gen // a success only: a newer failed read must not drop an older good one

"Newest wins" is the obvious rule for overlapping refreshes, and it has to mean newest successful. A failed read is not news about the host's models. It is news about the read.

What the journal says now#

The journal is where the truth goes. Its line says whether the file was applied or the running config kept, and why, down to the field that needs a restart. An applied reload also says that a running engine keeps its old preset until it next starts. And once the reload's own read of a host has succeeded, a periodic read that started before the reload can finish as late as it likes: its generation is older, and the proxy drops it.

systemctl reload still reports success every time, including the times the file didn't load, and it keeps its place at the top of the list. I haven't made my peace with that. I've made arrangements: the command delivers the signal, and the journal tells me what the signal did, on both clocks.

Filed under: go, linux, systemd, local-llm

Sources#