Cutting a stream doesn't cancel the turn

· 11 min read
Contents

The plan was a router. One OpenAI- and Anthropic-compatible URL on a mini-PC, two GPU boxes behind it, read the request's model, pick the box, forward the bytes. I started it at 23:37 one night. By 22:04 the next evening it was starting and stopping llama.cpp builds on the 3090 box (the A9's benchmarks kept it out that day), deciding when a request has to wait, and shutting engines down after fifteen idle minutes so a benchmark could have the GPU back.

Most of that was plumbing. One decision wasn't: at 01:42 I told it to switch engines the moment a request needed another one, even if that cut someone else's stream. At 16:06 I took it back. Cutting a stream doesn't cancel anything. The client just sends the turn again.

A model name that says which box#

The two boxes are an RTX 3090 with 24 GiB and the A9 Mega, a 128 GB Strix Halo mini-PC. Both serve models through llama.cpp's router mode, one model loaded at a time, and several models exist on both: Flint, K2, Laguna, Qwen.

The first routing commit, at 23:48, had a sticky host with failover: if the 3090 was down, send the request to the A9's copy. It lasted twenty minutes. The copies aren't the same model in any way that matters to a client: different quants (the A9 is meant for the higher ones), different builds (CUDA on one box, Vulkan or ROCm on the other), different measured context. A backup that answers differently is a second model with the first one's name.

So the host went into the name. A client asks for 3090-laguna-xs or a9-laguna-xs-q8, and the proxy strips the host from the top-level model value (laguna-xs, laguna-xs-q8) before it forwards the request. The rest passes through: llama.cpp's timings, /props?model=, /slots, native Anthropic /v1/messages. The few deliberate exceptions came later (credentials are dropped before a GPU box sees them, two error messages get reworded for Claude Code, and a keepalive I'll get to). If the 3090 is down, 3090-anything is unavailable, and the error says so.

One GPU, four engines#

"Several llama.cpp builds" is where routing stopped being enough. The 3090 box runs mainline llama.cpp, a newer llama.cpp commit for Flint and K2 (its engine is called fork, though it is upstream llama.cpp), a separate build for Bonsai, and Strata in a podman container. Each one wants the whole GPU, and the box's own serve scripts refuse to start while any other model server runs. A model name can be valid and still not be servable right now, because the wrong engine is up.

The proxy got a systemd --user unit per engine on each box (Conflicts= the others, never enabled, Restart=no), a small helper on the GPU side that is the same Go binary run over ssh with one JSON request on stdin, and a switch: a request for a key on a stopped engine stops the running engine and starts the right one. From the proxy's journal that morning:

2026/10/24 10:48:23 host 3090: switching to engine bonsai for 3090-bonsai2-27b
2026/10/24 10:48:26 host 3090: engine bonsai up after 2s
2026/10/24 10:48:31 POST /v1/chat/completions model="3090-bonsai2-27b" host=3090 status=200 bytes=674 dur=7.798s
2026/10/24 10:48:31 host 3090: switching to engine strata-q2_0 for 3090-flashnext-q2_0-strata
2026/10/24 10:49:37 host 3090: engine strata-q2_0 up after 1m6s
2026/10/24 10:49:38 POST /v1/chat/completions model="3090-flashnext-q2_0-strata" host=3090 status=200 bytes=676 dur=1m6.835s
2026/10/24 10:49:38 host 3090: switching to engine mainline for 3090-laguna-xs
2026/10/24 10:49:46 host 3090: engine mainline up after 8s

llama.cpp builds come up in 2 to 4 seconds (8 when mainline replaces Strata), and Strata in about a minute. (One switch that morning took 96 seconds, for a reason that has its own post.)

"Always, even if it cuts other streams"#

The 01:42 version of the rule went into the package comment in so many words: the proxy switches the host to the engine that has the key, "always, even if it cuts other streams". It fit the design's premise, one client active at a time on one LAN, and it was the shortest path to a working switch.

Then I captured what the coding agents I use do when a stream dies halfway, against a mock server that cuts it. opencode 1.18.4 sent the same turn again 2, 4, 8, 16 and then 30 seconds later before I stopped the capture, and its code says it would have kept going: its retry policy has no attempt limit, and a cut stream backs off to 30 seconds and stays there. pi 0.82.0 re-sent three times, 2, 4 and 8 seconds apart, then gave up with terminated.

Put two sessions on one box with models on different engines and the rule writes its own loop. Session A's request switches the box and cuts B. B re-sends, which switches the box back and cuts A. A re-sends. Each swing costs a switch, 2 to 4 seconds, or a minute when one side is Strata, and with opencode nobody ever stops. I never let that run live; the captures make it arithmetic. opencode doesn't even need a second session: its first turn sends a title request next to the real one, and a small_model on another engine of the same box would switch the box at the start of every session. The cut stream wasn't a cancellation. It was a retry with a delay.

Wait while busy, then hold for 30 seconds#

The 16:06 rule: a switch waits until the box has no request in flight, and an engine that just came up keeps the GPU for 30 seconds before anything can switch it away. Two refinements came out of review that evening. While a switch waits, new requests for the running engine now queue behind it, so a steady trickle can't starve it. Inside the 30-second hold-down they don't, so a fresh engine serves the clients that were waiting for it.

Two streams against the 3090 right after the change went live, one on mainline (laguna-xs), one on the Flint build (3090-custom-flint-40b is the same model as 3090-flint-40b; the middle word names the kind of engine):

2026/10/24 16:07:18 host 3090: switching to engine mainline for 3090-laguna-xs
2026/10/24 16:07:21 host 3090: engine mainline up after 2s
2026/10/24 16:07:33 POST /v1/chat/completions model="3090-laguna-xs" host=3090 status=200 bytes=368170 dur=14.605s
2026/10/24 16:07:33 host 3090: 3090-custom-flint-40b waits for the running engine before switching to fork
2026/10/24 16:07:51 host 3090: switching to engine fork for 3090-custom-flint-40b
2026/10/24 16:07:54 host 3090: engine fork up after 3s
2026/10/24 16:08:09 POST /v1/chat/completions model="3090-custom-flint-40b" host=3090 status=200 bytes=4473 dur=36.26s
2026/10/24 16:08:24 host 3090: 3090-laguna-xs waits for the running engine before switching to mainline
2026/10/24 16:08:26 host 3090: switching to engine mainline for 3090-laguna-xs
2026/10/24 16:08:28 host 3090: engine mainline up after 2s
2026/10/24 16:08:28 host 3090: 3090-custom-flint-40b waits for the running engine before switching to fork
2026/10/24 16:08:46 POST /v1/chat/completions model="3090-laguna-xs" host=3090 status=200 bytes=364227 dur=22.471s
2026/10/24 16:08:58 host 3090: switching to engine fork for 3090-custom-flint-40b
2026/10/24 16:09:02 host 3090: engine fork up after 3s
2026/10/24 16:09:17 POST /v1/chat/completions model="3090-custom-flint-40b" host=3090 status=200 bytes=4556 dur=48.723s

The second round is the interesting one. The Flint request arrived at about 16:08:28, the second the Laguna request ahead of it got its engine and began to stream; that stream finished at 16:08:46, status 200, 364,227 bytes, untouched. The switch happened at 16:08:58, exactly 30 seconds after mainline had come up. In the first round the Flint request arrived as the first Laguna stream ended and waited out the hold-down: mainline up at 16:07:21, switch at 16:07:51.

A wait can be much longer than 30 seconds: behind a long stream, or for Strata, up to the 15-minute wait budget. With nothing on the wire, a client with a 300-second header timeout gives up and re-sends, which is the loop again with extra steps. So after 10 seconds of waiting, a streaming request gets the proxy's own 200, text/event-stream and an SSE comment every 15 seconds until the real stream starts (what that costs, and the rest of what agent clients expect).

Waiting has its own edge: a client that gives up while its switch is still queued shouldn't move the GPU. An independent re-review caught an earlier version doing exactly that, switching the box to an engine nobody was waiting for any more and then back. Now a switch whose client has left before the start is abandoned. That review found bugs in every pass until the last, mostly in the fixes.

Fifteen minutes, then the GPU goes back#

Both boxes run benchmark queues, and those were there first. The 3090's benchmark guard refuses to start while any llama-server runs; the A9's job runner owns its GPU most of the time. The proxy never stops anything it didn't start. A benchmark unit, a GPU owner file held by someone else, a stray llama-server outside its units: any of those turns a switch into a refusal saying the GPU is busy (a 423 now, and not a 503 for a reason), and the request goes nowhere.

The other half is idle stop. After idle_stop (15 minutes by default) with nothing in flight, the proxy stops its own engine unit, and the box is free for whatever was queued. The end of the full test cycle that morning:

2026/10/24 11:33:32 POST /v1/chat/completions model="3090-strata-flashnext-q2_0-strata" host=3090 status=200 bytes=14098 dur=1.018s
2026/10/24 11:45:10 host a9: idle 15m0s, engine stopped
2026/10/24 11:48:46 host 3090: idle 15m0s, engine stopped

The 3090's last request ended at 11:33:32 and its engine stopped at 11:48:46, 15 minutes 14 seconds later. The A9 line is a bug. The A9's bench owned its GPU every time the proxy asked that day, and the only request for it in that cycle, at 11:29:43, had been refused (still a 503 at that hour):

2026/10/24 11:29:43 host a9: switching to engine mainline for a9-laguna-xs-q8
2026/10/24 11:29:43 POST /v1/chat/completions model="a9-laguna-xs-q8" host=- status=503 bytes=251 dur=184ms

The switch code had marked the A9 as active before asking it to start anything, so 15 minutes later idle stop dutifully sent a stop to a box where the proxy had never started an engine. It stopped nothing; the helper only touches the proxy's own units. It had done the same at 11:09:40, fifteen minutes after the service started, with no request for the A9 at all. It was still a stop call to a box running somebody's benchmark. The fix landed at 11:47, two minutes after the line: activity counts once an engine is up, and idle stop skips hosts that are down.

Gotcha"No route to host" from a box that answers ping

Until 09:48 the proxy logged the A9 as unreachable, while ping and ssh to it worked fine:

2026/10/24 09:44:29 host a9 down: Get "http://192.0.2.30:8080/health": dial tcp 192.0.2.30:8080: connect: no route to host

firewalld's default zone target accepts ICMP and rejects everything else, and its nftables backend rejects with an ICMP admin-prohibited, which Linux turns into EHOSTUNREACH. After a rich rule that lets only the proxy host reach port 8080, the same poll said what was true, that the port was reachable and nothing listened on it:

2026/10/24 10:28:38 host a9 down: Get "http://192.0.2.30:8080/health": dial tcp 192.0.2.30:8080: connect: connection refused

Most of what the proxy knows now is a list of things it won't do: answer from another box's copy, cut a running stream, touch a GPU it didn't start. The first version of the switching rule lasted 14 hours and 25 minutes. More than half of it was the night in between.

Filed under: local-llm, llama.cpp, go, systemd, proxy

Sources#