The proxy said unloaded. The model was loading.

Ten times in sixteen seconds the probe asked my proxy for its model list, and ten times it got back the same 54,031 bytes, with the model it had just asked to load listed as unloaded. After fifteen seconds of that, the probe wrote its verdict and moved on:

{"alias": "3090-flint-40b", "configured_context": 131072, "result": "error", "error": "3090-flint-40b returned to unloaded during load", "total_s": 18.4}

Four of the seven models in that run got the same line. None of them had unloaded. The 3090 box's router had accepted each load, exactly as asked.

The proxy is the LAN endpoint in front of my two GPU boxes, the one that switches llama.cpp engines without cutting streams. The sweep was the slowest check on its verification list: the 3090 box's own context probe, the script that proves each model serves the context it claims, pointed at the proxy instead of the box. A 261,488-token prompt takes minutes to prefill. The first pass over seven models took seventeen of them, and the four failures were the quick part: under a minute and a half between them. Direct, the probe passes every one of these models.

The probe gives a load fifteen seconds#

The probe asks llama.cpp's router to load a model, then polls /models every two seconds until the model says loaded:

def status(alias):
    return next(m['status']['value'] for m in request('/models')['data'] if m['id'] == alias)


def wait_status(alias, wanted, seconds):
    started = time.monotonic()
    current = None
    while time.monotonic() - started < seconds:
        current = status(alias)
        if current in wanted:
            return current
        if current == 'unloaded' and time.monotonic() - started > 15 and 'loaded' in wanted:
            raise RuntimeError(f'{alias} returned to unloaded during load')
        time.sleep(2)
    raise TimeoutError(f'{alias} stayed {current} for {seconds}s')

The unloaded branch is the one that fired. Against a llama.cpp router it's the right rule. The router lists a model as loading while it loads, so a model that still reads unloaded fifteen seconds after its load request was accepted has died on the way. Waiting out the full ten minutes on a dead load would be ten minutes nobody gets back.

Through the proxy, though, "the router says" had become "the proxy says the router said".

Ten answers, one snapshot#

The proxy's journal for the first failure, 3090-flint-40b:

2026/10/24 22:21:39 GET /health model="" host=- status=200 bytes=64 dur=0s from=192.0.2.10 ua="Python-urllib/3.14"
2026/10/24 22:21:39 GET /models model="" host=- status=200 bytes=49489 dur=1ms from=192.0.2.10 ua="Python-urllib/3.14"
2026/10/24 22:21:39 host 3090: switching to engine fork for 3090-flint-40b
2026/10/24 22:21:41 host 3090: engine fork up after 2s
2026/10/24 22:21:41 POST /models/load model="3090-flint-40b" host=3090 status=200 bytes=16 dur=2.326s from=192.0.2.10 ua="Python-urllib/3.14"
2026/10/24 22:21:41 GET /models model="" host=- status=200 bytes=54031 dur=1ms from=192.0.2.10 ua="Python-urllib/3.14"
2026/10/24 22:21:43 GET /models model="" host=- status=200 bytes=54031 dur=1ms from=192.0.2.10 ua="Python-urllib/3.14"
2026/10/24 22:21:45 GET /models model="" host=- status=200 bytes=54031 dur=1ms from=192.0.2.10 ua="Python-urllib/3.14"
2026/10/24 22:21:47 GET /models model="" host=- status=200 bytes=54031 dur=1ms from=192.0.2.10 ua="Python-urllib/3.14"
2026/10/24 22:21:49 GET /models model="" host=- status=200 bytes=54031 dur=2ms from=192.0.2.10 ua="Python-urllib/3.14"
2026/10/24 22:21:51 GET /models model="" host=- status=200 bytes=54031 dur=1ms from=192.0.2.10 ua="Python-urllib/3.14"
2026/10/24 22:21:53 GET /models model="" host=- status=200 bytes=54031 dur=2ms from=192.0.2.10 ua="Python-urllib/3.14"
2026/10/24 22:21:55 GET /models model="" host=- status=200 bytes=54031 dur=1ms from=192.0.2.10 ua="Python-urllib/3.14"
2026/10/24 22:21:57 GET /models model="" host=- status=200 bytes=54031 dur=1ms from=192.0.2.10 ua="Python-urllib/3.14"
2026/10/24 22:21:57 GET /models model="" host=- status=200 bytes=54031 dur=1ms from=192.0.2.10 ua="Python-urllib/3.14"
2026/10/24 22:21:58 POST /models/load model="3090-flint-40b-256k" host=3090 status=200 bytes=16 dur=423ms from=192.0.2.10 ua="Python-urllib/3.14"
2026/10/24 22:21:58 GET /models model="" host=- status=200 bytes=54031 dur=1ms from=192.0.2.10 ua="Python-urllib/3.14"
2026/10/24 22:22:00 GET /models model="" host=- status=200 bytes=54045 dur=2ms from=192.0.2.10 ua="Python-urllib/3.14"

The load itself worked. The proxy switched the box to the engine that serves Flint, passed the request on and got a 200 in 2.3 seconds. Then came the probe's nine polls and the one last status check it makes before moving on, and the proxy answered all ten without asking the box. Its /models is a merged list built from a snapshot of each box's own list, and in v0.2.0 nothing refreshed that snapshot after a load went through. The next 30-second poll did. The size moved for the first time at 22:22:00, nineteen seconds after the load request returned and three seconds after the probe had given up on it.

So pass or fail came down to where in the poll cycle a load happened to start. The next key, 3090-flint-40b-256k, sent its load request at 22:21:58, two seconds before that refresh, and passed. The four failures fit the same arithmetic. Each took its own load request plus about sixteen seconds, the first two-second poll past the fifteen-second limit:

The best part is that the proxy could have known. llama.cpp's router publishes model state changes as events on /models/sse, and since v0.2.0 the proxy followed that feed for every box running a router engine, to merge the events into a feed of its own for clients:

for sc.Scan() {
	data, ok := strings.CutPrefix(sc.Text(), "data: ")
	if !ok {
		continue
	}
	var ev map[string]any
	if json.Unmarshal([]byte(data), &ev) != nil {
		continue
	}
	key, _ := ev["model"].(string)
	if key == "" || key == "*" {
		ev["model"] = host + "-*"
	} else {
		ev["model"] = host + "-" + key
	}
	ev["proxy"] = map[string]any{"host": host, "key": key}
	p.events.broadcast(ev)
}

Rename the model, add a proxy field, broadcast. Every event the proxy received this way went out, relabelled, to every subscriber, and the snapshot that /models answered from never heard any of them. The proxy passed the news along to everyone except itself. It didn't always get the news, either: three of the four failed loads came right after an engine switch, and an event sent before the proxy has reconnected to the new engine's feed isn't replayed.

The test copy nearly took the fix#

The probe in the sweep was a copy, with the base URL pointed at the proxy and a longer timeout on the load request, since a load through the proxy may have to switch engines first. Two minutes into the first run, the copy's fifteen-second limit also became sixty, with a comment that blamed the proxy's lagging /models.

Sixty seconds outlasts a 30-second poll, so every model would have passed, and a correct rule in the client would have become a slower one, to cover for a bug in the server. It never decided a result. The running probe had loaded its module before the edit. The only later probe in that run was Strata's, which takes the direct path and never waits on a status. The reruns used a fresh copy with the original fifteen seconds. The fix went into the proxy.

Re-poll when the box says something changed#

v0.2.1 was serving by 22:43 that night. The proxy now re-reads a box's list after every load or unload it passes through, whenever that box's event feed reports a change (download progress aside, at most once every two seconds), and once each time it connects to the feed, for the events it missed. The first of the three, in serveTo:

@@ -444,6 +444,10 @@ func (p *Proxy) serveTo(w http.ResponseWriter, r *http.Request, rec *recorder) (
 		w = uw
 	}
 	p.proxies[host].ServeHTTP(w, r)
+	if (r.URL.Path == "/models/load" || r.URL.Path == "/models/unload") && rec.status < 300 {
+		// The router changed a model's state: /models shows it now, not at the next periodic poll.
+		go p.cat.PollHost(context.Background(), host)
+	}
 	return model, host
 }
 

The same kind of load after the fix, 3090-flint-UD-Q3_K_XL:

2026/10/24 22:44:43 GET /models model="" host=- status=200 bytes=54039 dur=2ms from=192.0.2.10 ua="Python-urllib/3.14"
2026/10/24 22:44:43 POST /models/load model="3090-flint-UD-Q3_K_XL" host=3090 status=200 bytes=16 dur=3ms from=192.0.2.10 ua="Python-urllib/3.14"
2026/10/24 22:44:43 GET /models model="" host=- status=200 bytes=54039 dur=2ms from=192.0.2.10 ua="Python-urllib/3.14"
2026/10/24 22:44:45 GET /models model="" host=- status=200 bytes=54045 dur=1ms from=192.0.2.10 ua="Python-urllib/3.14"
2026/10/24 22:44:47 GET /models model="" host=- status=200 bytes=54045 dur=1ms from=192.0.2.10 ua="Python-urllib/3.14"
2026/10/24 22:44:49 GET /models model="" host=- status=200 bytes=54045 dur=2ms from=192.0.2.10 ua="Python-urllib/3.14"
2026/10/24 22:44:51 GET /models model="" host=- status=200 bytes=54045 dur=2ms from=192.0.2.10 ua="Python-urllib/3.14"
2026/10/24 22:44:53 GET /models model="" host=- status=200 bytes=54045 dur=2ms from=192.0.2.10 ua="Python-urllib/3.14"
2026/10/24 22:44:55 GET /models model="" host=- status=200 bytes=54309 dur=2ms from=192.0.2.10 ua="Python-urllib/3.14"
2026/10/24 22:44:55 GET /props model="3090-flint-UD-Q3_K_XL" host=3090 status=200 bytes=4275 dur=3ms from=192.0.2.10 ua="Python-urllib/3.14"

The probe's first poll, in the same second as the load request, still got the old list: the re-read runs after the response has gone out, and the poll won that race. The second poll, two seconds later, got the new one, and the list changed again when the model was up. The four failed keys, rerun through v0.2.1 with the probe's own fifteen-second rule:

Every one within four tokens per second of direct. Two of the reruns first got a 400 from llama.cpp itself, model is already running. Run again from a clean state, both passed.

A 336 ms stream that took 120 seconds#

The last check on the list was the cheap one: does the first SSE line of a stream arrive as fast through the proxy as it does direct? The same small streaming request, alternating between the box and the proxy. The first run, whose log ends after three rounds:

== step 7 23:45:27
warm-up via proxy (starts the engine, loads the model): (200, 5.892084068036638, 126.13260120700579)
warm-up direct: (200, 0.16560008196393028, 0.363123552990146)
run 0: direct 200 first 137.7 ms total 335 ms | proxy 200 first 137.2 ms total 120333 ms
run 1: direct 200 first 165.3 ms total 363 ms | proxy 200 first 136.7 ms total 120335 ms
run 2: direct 200 first 164.8 ms total 359 ms | proxy 200 first 137.9 ms total 120336 ms

The first line came through the proxy no slower than direct in any of the three rounds. The stream ended two minutes later. Those 120 seconds belonged to the test. It read lines from r.fp, the raw socket under Python's HTTPResponse, and stopped at end of file. Direct, that happens to work: the box ends a streaming response by closing the connection, so the end of the stream and the end of the file arrive together, about a third of a second in.

Through the proxy they don't, because the client's connection belongs to the proxy. Connection and the other hop-by-hop headers stop at the first hop: RFC 9110 tells intermediaries to remove them before forwarding, and Go's httputil.ReverseProxy strips them from the upstream response before copying the rest to the client. However the box ends its stream, the proxy's own server decides what happens to the client's connection, and it does what an HTTP/1.1 server does with a finished response: it keeps the connection open for the next request, here until IdleTimeout: 2 * time.Minute. The response had ended on time, with chunked encoding's empty last chunk, but a reader of raw lines sees that terminator as a couple more lines and keeps waiting for a close that comes two minutes later. CPython warns about the same trap, for a response with a Content-Length, in a comment at the top of HTTPResponse.__init__:

# If the response includes a content-length header, we need to
# make sure that the client doesn't read more than the
# specified number of bytes.  If it does, it will block until
# the server times out and closes the connection.  This will
# happen if a self.fp.read() is done (without a size) whether
# self.fp is buffered or not.  So, no self.fp.read() by
# clients unless they know what they are doing.
self.fp = sock.makefile("rb")

A reproduction on localhost, with a toy upstream that closes the connection after three events and a [DONE] and a Go reverse proxy in front of it whose idle timeout is five seconds instead of two minutes:

$ python3 client.py
direct r.fp.readline  events=4  total=0.305 s
direct r.readline     events=4  total=0.303 s
proxy  r.fp.readline  events=4  total=5.307 s
proxy  r.readline     events=4  total=0.302 s

The fixed test reads with r.readline(), which goes through the response's chunked decoding and reports the end of the stream when the last chunk arrives. Eight alternating runs after a warm-up:

== step 7 23:55:45
warm-up via proxy (starts the engine, loads the model): (200, 0.14075446198694408, 0.3378924409626052)
warm-up direct: (200, 0.13881341298110783, 0.3374847459490411)
run 0: direct 200 first 137.7 ms total 336 ms | proxy 200 first 135.6 ms total 336 ms
run 1: direct 200 first 136.3 ms total 335 ms | proxy 200 first 136.1 ms total 334 ms
run 2: direct 200 first 138.4 ms total 337 ms | proxy 200 first 136.8 ms total 337 ms
run 3: direct 200 first 136.3 ms total 337 ms | proxy 200 first 139.0 ms total 340 ms
run 4: direct 200 first 137.4 ms total 337 ms | proxy 200 first 136.1 ms total 335 ms
run 5: direct 200 first 137.2 ms total 334 ms | proxy 200 first 136.6 ms total 336 ms
run 6: direct 200 first 136.0 ms total 335 ms | proxy 200 first 135.6 ms total 336 ms
run 7: direct 200 first 137.2 ms total 336 ms | proxy 200 first 138.1 ms total 336 ms
median first SSE line: direct 137.2 ms, proxy 136.3 ms, difference -0.9 ms
== done 23:55:51

The proxy came out 0.9 ms faster than the box it forwards to, which is noise, and which I have decided not to enjoy.

The probe still gives a load fifteen seconds. Through the proxy, a load now shows up within two.

Filed under: local-llm, llama.cpp, go, proxy, testing

Read next#

• Cutting a stream doesn't cancel the turn — what the proxy is, and why a load through it may wait for an engine switch.
• To a coding agent, 503 means try again — the early 200 and the keepalive comments the proxy sends while a client waits.
• The fixes had the bugs — seven review passes over the same proxy, the same evening.
• The design review found a Konsole tab — what a design review found later that night, in parts of the proxy that worked.

Sources#

• llama.cpp server README — router mode: the status values unloaded, loading, loaded; POST /models/load; the GET /models/sse event feed.
• CPython Lib/http/client.py (3.14) — HTTPResponse.fp = sock.makefile("rb"), the comment about self.fp.read(), and readline() falling back to chunk-aware reads.
• Go net/http/httputil/reverseproxy.go (release-branch.go1.26) — hopHeaders, removeHopByHopHeaders(res.Header).
• Go net/http/server.go (release-branch.go1.26) — Server.IdleTimeout.
• RFC 9110, section 7.6.1 — intermediaries remove the Connection header field before forwarding.