The design review found a Konsole tab

· 10 min read
Contents

Before I read the design review of my LAN proxy, three reports from three sides, I wrote down my own, so that afterwards I couldn't pretend to have seen anything coming. Three findings, four positions I'd defend. The first finding was a real one: every request body went through a JSON parser three or four times before the proxy forwarded it, about 30 ms for a 1.4 MB request by my estimate. The performance review measured 20.5 ms for a 1.2 MB body through the whole proxy, against 0.48 ms direct, and rated it MEDIUM.

None of the reports' HIGH findings was on my list. Several were in things I hadn't listed because they worked.

The proxy sits on a mini-PC in front of two GPU boxes, routes OpenAI- and Anthropic-style requests by model name, and switches llama.cpp engines on the boxes over ssh when a request needs another one (how, and why it waits). By that night its fixes had been through seven review passes, and a list of live checks had passed through it, a 262k-token context sweep among them. The design review challenged the design itself: performance, architecture and usability.

Engine control lived in a Konsole tab#

The proxy runs its own binary on each GPU box as an agent: ssh <box> .local/bin/llm-router-proxy agent, one JSON request on stdin, one reply. In v0.2.2 that call came with exactly two ssh options:

args := append([]string{"-o", "BatchMode=yes", "-o", "ConnectTimeout=5", target}, command...)

Everything else came from ~/.ssh/config, where Host * sets ControlMaster auto and ControlPersist 10m. The first ssh connection to a box becomes a master, and later ones go through its socket instead of logging in again.

The architecture review tried a fresh login from the proxy host to the 3090 box, with the master bypassed, and got Permission denied (publickey,…). The box's authorized_keys held two ed25519 keys, and neither one was the proxy host's. The same login to the A9 worked. So on the 3090, every engine switch, idle stop and roster read had been going through one master connection, opened on 22 October, whose process sat, according to its cgroup, inside a Konsole tab. The project's brief still said the proxy host's key logged in to that box.

Nothing in the proxy would have noticed until the tab closed, or the box rebooted, or the link dropped for long enough. From then on every agent call to the 3090 fails. A switch answers 424 with the ssh error inside, idle stop fails and retries every 30 seconds, and the rosters, the per-engine lists of models the proxy reads over ssh every ten minutes, keep their last good copy until the proxy restarts. After that, the 3090 has no roster at all, and while its engine is idle-stopped, every 3090-* name gets a 424 saying the host is down, without the proxy even trying to start it. That chain is read from the code. Nobody closed the tab to check.

The live checks that switched engines that day had all passed through that master, which is part of why they passed. Things that work because of the machine you're sitting at look exactly like things that work. My list of things that work by accident got a new first entry: a terminal tab, open since Thursday.

v0.3.0, and a key installed in both boxes' authorized_keys, give the proxy an identity of its own: ~/.config/llm-router-proxy/id_ed25519, offered alone, with a master socket under $XDG_RUNTIME_DIR and keepalives of its own (every 5 seconds, three misses), so a dead link becomes a failed call instead of a hung one:

func sshArgs(target, key, runtimeDir string) []string {
	args := []string{"-o", "BatchMode=yes", "-o", "ConnectTimeout=5", "-o", "ServerAliveInterval=5", "-o", "ServerAliveCountMax=3"}
	if _, err := os.Stat(key); err == nil {
		args = append(args, "-i", key, "-o", "IdentitiesOnly=yes")
		// A socket path must fit sun_path (108): dir + "/lrp-" + 40 hex (%C) + ssh's 17-char temp suffix.
		if len(runtimeDir)+5+40+17 < 108 {
			args = append(args, "-o", "ControlMaster=auto", "-o", "ControlPersist=10m", "-o", "ControlPath="+filepath.Join(runtimeDir, "lrp-%C"))
		} else {
			args = append(args, "-o", "ControlMaster=no", "-o", "ControlPath=none")
		}
	}
	return append(args, target)
}

/status now shows per box whether engine control works, from a real agent call rather than from hope.

help started a proxy#

The binary has five jobs: the proxy itself, plus the engine, model, client and agent subcommands. v0.2.2 checked the first argument for those four words and handed everything else to the flag parser:

if len(os.Args) > 1 && os.Args[1] == "engine" {
	os.Exit(engineMain(os.Args[2:]))
}
if len(os.Args) > 1 && os.Args[1] == "client" {
	os.Exit(clientMain(os.Args[2:]))
}
if len(os.Args) > 1 && os.Args[1] == "model" {
	os.Exit(modelMain(os.Args[2:]))
}
if len(os.Args) > 1 && os.Args[1] == "agent" { // GPU-host side, run over ssh by the proxy
	os.Exit(agent.Main(os.Args[2:], os.Stdin, os.Stdout, os.Stderr))
}
configPath := flag.String("config", defaultConfigPath(), "hosts config (JSON)")
listen := flag.String("listen", "", "override the config's listen address")
showVersion := flag.Bool("version", false, "print version and exit")
check := flag.Bool("check", false, "validate the config and exit")
flag.Parse()

The flag package documents what happens next: "Flag parsing stops just before the first non-flag argument". help is a non-flag argument. flag.Parse() stops there, returns without an error, nothing ever looks at what's left in flag.Args(), and the proxy starts with the live config. So did status, and models status with one letter too many.

The usability review found it by doing it. At 00:04:06, a help typed to read the usage text started a full proxy against the live config. It polled both boxes and then died on bind: address already in use, because the real service held the port. That was the lucky version. The project's procedure for smoke runs was to stop the service first, and with the service stopped the same typo is a second proxy switching and idle-stopping the real engines, which the project's rules forbid in so many words. v0.3.0 makes any leftover word a usage error:

if fs.NArg() > 0 {
	fmt.Fprintf(stderr, "llm-router-proxy: unknown command %q\n", fs.Arg(0))
	fs.Usage()
	return o, 2
}

The hold-down started before the model did#

The 30-second hold-down is what keeps two clients on two engines from flipping a box back and forth (where it came from): after a switch, the new engine is safe from the next one for half a minute. The clock started when the new engine listed the requested model. A llama.cpp router lists every model it has as soon as it's up, and loads one only when the first request for it arrives. From the proxy's journal that night, a switch to fork, the llama.cpp build that serves K2:

2026/10/24 23:43:16 host 3090: switching to engine fork for 3090-k2-horizon-36b
2026/10/24 23:43:20 host 3090: engine fork up after 4s
2026/10/24 23:43:36 POST /v1/chat/completions model="3090-k2-horizon-36b" host=3090 status=200 bytes=885 dur=19.402s in=195 out=36 tok/s=45.29513012643812 from=192.0.2.10 ua="Python-urllib/3.14"

Engine up at 23:43:20, first answer at 23:43:36, the request's 19.4 seconds mostly spent loading the model. Sixteen of the thirty seconds were gone before the first answer, spent protecting a request that the rule about waiting for in-flight requests was already protecting. On the A9, whose models are bigger and take minutes to load, the hold-down is over before the first answer every time.

The cost shows with two clients. The architecture review ran the switch code with the clock scaled down (300 ms of hold-down, a 400 ms load), a client B taking three turns on one engine and a client A waiting for the other (fork and mainline, two of the box's llama.cpp builds):

Hold-down clockEngine startsB's three turnsA's one request
starts when the model is listed (v0.2.2)fork, mainline, fork560, 1111, 51 ms968 ms
starts at the first answer (6-line patch)fork, mainline558, 51, 51 ms1265 ms

With the clock running from the first answer, the waiting switch can't take the GPU between B's turns, and B's second turn costs 51 ms instead of 1111. A waits longer, which is the trade the hold-down exists to make. I've approved the change, as part of a per-host scheduler rewrite.

The same code had a floor under every start. The performance review counted the llama.cpp engine starts in the journal for 24 October: of 31, 12 logged up after 2s, and two seconds was as fast as it got, because after starting an engine the proxy slept one fixed two-second poll before its first look.

Four JSON parses per request: the finding I did write down#

My own top finding survived with less weight. The cost per megabyte was about what I'd estimated; its share of a turn wasn't. The proxy parsed each body four times, not three or four: once to find model, twice to see whether stream was set, once more to rewrite model, and it allocated about 8.4 bytes per byte of body on the way. Through the proxy to a fake box that reads the body, against the same box direct:

BodyDirectThrough the proxy
1.2 MB0.48 ms20.5 ms
5.8 MB2.1 ms95.5 ms

A 22 MB body with a base64 image took about 445 ms in the parsing alone. On a coding agent's warm turn, which resends the whole history and gets most of it from the prompt cache, the parse is a few percent of the time to the first token, on every turn, not the five to twenty percent I had guessed. MEDIUM is the right word. The fix comes in steps: the first is one line (isStream once: 123 ms down to 93 in the parse benchmark at 5.8 MB), and the last, a one-pass parser at about 20 ms for the same body, waits until image-heavy bodies are normal, and then for a fuzz test against the current code before it goes anywhere near the byte-exact path.

My four positions did better than the finding. The first, that users feel switch latency and the idle stop rather than the proxy's own overhead, is where the performance review's numbers landed: a few percent of a turn for the parse, most of a router switch for the two-second poll. The other three the design review kept outright. One ssh call per agent operation costs about 60 ms on the box and 70 ms of ssh, against engine starts of 2 to 8 seconds. Never answering 503 is what keeps coding agents from retrying forever. State stays in memory, because the boxes know what's running. Byte-exact forwarding stayed too, and so did the streaming path, on the first-SSE-line parity measured live that same night: 137.2 ms direct and 136.3 ms through the proxy. The design review also found that the five-second shutdown grace was dead code, which got a post of its own.

Both boxes take the proxy's own key now, and at one the next afternoon /status.json reported engine control ok for both. The Konsole tab can close whenever it likes.

Filed under: go, ssh, design, proxy, code-review

Sources#