The first engine on the A9 was started to be stopped

Stopping a process is the easy half of proving that my LLM proxy gives a GPU back. On the A9 the hard half was letting it take the GPU at all. The one rule I keep about that box is that nothing gets in its benchmark's way, and the proxy had never started an engine there. The day before, its idle stop had sent the A9 a stop anyway (how idle stop managed that).

At 16:56:49 on 25 October the proxy started its first engine on the A9, on purpose, so that a benchmark job could take the GPU away from it. The engine was up for 47 seconds, which was the whole idea.

With an old agent, every start is refused#

The release that made this possible, v0.3.1, shipped at 15:16. What it does about the A9's benchmark is in the post before this one: a job in the queue's running/ and the two minutes after running/ changes count as the benchmark's, and when the runner logs that a gpu job waits behind the proxy's engine, the proxy stops that engine as soon as nothing runs on it. Reading the queue and the runner's log is the job of the agent, the proxy's own binary on each GPU box, which the proxy runs over ssh. Without that reading the proxy has nothing to act on, and an agent from v0.3.0 takes the new job_queue setting the way it takes any field it doesn't know: it decodes the request, drops the field and answers as if nothing had been asked.

So the v0.3.1 proxy asks. The agent's status reply says whether it read the queue, and on a box that has one, a reply that doesn't say so counts as someone else holding the GPU:

// Status reads which engine unit runs on host and who else holds the GPU.
func (m *Manager) Status(ctx context.Context, host string) (Status, error) {
	h, err := m.host(host)
	if err != nil {
		return Status{}, err
	}
	rep, err := m.call(ctx, h, agent.Request{Op: "status", Port: port(h), BenchPort: BenchPort, JobQueue: h.JobQueue})
	st := Status{Active: rep.Active, State: rep.State, Foreign: rep.Foreign, BenchWaits: rep.BenchWaits}
	if err == nil && h.JobQueue != "" && !rep.QueueChecked { // an older agent ignores job_queue: fail closed
		st.Foreign = append(st.Foreign, oldAgent(h))
	}
	return st, err
}

// oldAgent says that host's agent predates job_queue, so the bench there is not protected.
func oldAgent(h config.Host) string {
	return fmt.Sprintf("the agent on host %s predates job_queue: run `llm-router-proxy engine install %s`", h.Name, h.Name)
}

The new agent reached the 3090 box at 15:16, the proxy itself was reinstalled at 15:16:46, and at 15:17:08 the A9's job_queue went into the live config while the A9 still ran the old agent. At 15:17 engine status a9 listed the benchmark's own llama-server, its port, and the oldAgent line. Until the agent was replaced, any engine start on the A9 would have got a 423, gaps between jobs included, where v0.3.0 had seen the benchmark's server while a job ran and nothing at all between two jobs. The journal shows no request for an A9 model in the 77 minutes that followed, except one test with a made-up model name, which got the 404 it deserved.

The A9 was in the middle of a benchmark batch, and its side asked for the install to wait until the batch was done. It waited. The batch's last job left running/ at 16:32:54. At 16:33:53 engine install a9 wrote one file and started nothing, and a status call a second later said the GPU was busy: the queue's running/ had changed 1m0s ago, and the runner might start its next job. That was the two-minute gap rule reading the real queue for the first time, exactly a minute after the batch's last job. At 16:35:08 it reported no owner at all.

A test job in the A9's own queue#

That left the live check, and the live check had a shape problem. It had to show the proxy giving the GPU back to a benchmark job that waited behind the proxy's engine. For that, a benchmark job had to wait behind the proxy's engine, which is the one state the A9's rule exists to prevent.

So the waiting job belonged to the benchmark side, and so did the timing. The check ran only after the A9's side confirmed its own testing was finished, and the job went into the A9's queue from that side: one gpu-lane job whose whole body was sleep 5, kept out of the seed script so that no re-seed ever puts it back. The proxy never touched the queue. I did what any client would and asked the proxy for a model (llm-router-proxy model load), the smallest the A9 offers through it, at 36 GB. The engine was up two seconds after the switch began, and by 16:57:01 the model was loaded: a router process with its model server as a child.

Four seconds from gpu-wait to engine stopped#

The proxy rows come from the proxy's journal, which a small watcher copied once a second during the check, except the 16:57:01 load and the two pids, which are from my notes of the check, and the two probes' reasons, which are their own responses, pasted below. The A9 rows are what the A9's side reported, the runner's events among them:

Two of the A9 rows have a check on the proxy's side: the proxy's own log line names the gpu-wait's 16:57:34, and the second probe's "34s ago" puts the done at 16:57:54. The 16:57:25 and 16:57:44 rows rest on the A9's report alone.

The two journal lines from 16:57:38, as the watcher copied them (its own clock first):

16:57:39 LOG 2026/10/25 16:57:38 host a9: bench job 00-gpu-proxy-handback-test.job waits for the GPU since 2026-10-25 16:57:34 (held by pid 2530256): refusing new work, the engine stops once idle
16:57:39 LOG 2026/10/25 16:57:38 host a9: bench job 00-gpu-proxy-handback-test.job waits for the GPU since 2026-10-25 16:57:34 (held by pid 2530256): engine stopped

Four seconds from the runner's line to the proxy's was the timing of one tick, not a property of the code. The idle tick that polls the bench runs every 30 seconds, and this one happened to come four seconds after the event. The same gpu-wait could have waited most of half a minute for it. The stop took under a second. It could run at once because nothing was in flight. A stream in progress would have run to its end first: the release stop never cuts one.

"Engine stopped" is checked now#

The second of those lines is a claim, and before v0.3.1 it was a cheap one. The agent's stop reported success once it had stopped the engine units it could see, whatever still ran in a cgroup it skipped: a unit that had failed with its process stuck after the final SIGKILL, or another user's llm-engine-* unit that the substring check of the time counted as ours. At 13:59 that would have become a loop. A fix for another case, an engine started outside the proxy after an idle stop, had removed the guard that capped the release stop at one, so each 30-second tick would still see the process the runner named, stop again, log "engine stopped", tell every /models/sse listener the engine had stopped, and open a fresh 30-second handover. A test showed it, and the fix went in at 14:27, before v0.3.1 shipped: no real runner ever met it. Since v0.3.1 the stop looks before it says anything:

// Stop stops every running engine unit. A GPU process still in an engine unit's cgroup afterwards (stuck after the
// final SIGKILL, or a unit list-units does not show) is an error: the caller must not report the GPU as free.
func (a *Agent) Stop() error {
	if err := a.stopExcept(""); err != nil {
		return err
	}
	for i := 0; ; i++ {
		ours, _ := a.gpuProcesses()
		if len(ours) == 0 {
			return nil
		}
		if i == 10 {
			return fmt.Errorf("%s pid %s of an engine unit still runs after the stop", ours[0].comm, ours[0].pid)
		}
		a.Sleep(time.Second)
	}
}

A stop that fails now logs no "engine stopped", sends no event and opens no handover. The tick retries it while the host is up, logs the failure once per failing streak, and shows it on /status as stop_error until a stop succeeds or the host is seen down. So the line at 16:57:38 means the agent looked after the stop and found nothing of ours left on the GPU. The A9's side looked too, afterwards: llm-engine-mainline inactive, no llama-server.

Two probes, two different refusals#

The watcher sent exactly one request of its own, and only while the handover held, so that it couldn't start the engine again. It went out at 16:57:39, a second after the stop:

16:57:39 PROBE during the hand-over: 423 {"error":{"code":423,"message":"model a9-ornith15-35b-q8: host a9: GPU is busy: handed to a waiting bench job","type":"gpu_busy"}}

That probe sat in the window the handover exists for. The runner doesn't take the GPU the moment the proxy lets go. It looks at its next tick, and that came at 16:57:44, five seconds later. Without the handover, the probe would have found the blocker gone and running/ unchanged, the agent would have called the GPU free, and the proxy would have started its engine again in front of the job (the race, and the 30 seconds that close it).

I sent the second probe by hand at 16:58:28, twenty seconds after the handover ended at 16:58:08:

{"error":{"code":423,"message":"model a9-ornith15-35b-q8: switching host a9 to engine mainline failed: host a9: GPU is busy, not switching: bench job queue ~/jobs/running changed 34s ago (the runner may start its next job)","type":"gpu_busy"}}

This time the proxy had no objection of its own. Its journal shows it starting to switch the A9 to mainline, and the agent refusing: running/ had changed 34 seconds earlier, when the test job left it, and the runner might be about to start the next one. Nothing started.

Fifty-four seconds from the gpu-wait to that answer, and the real queue had walked the proxy through three of the holes that day's commits closed: a job waiting behind the proxy's engine, the race in the handover, and the gap between two jobs. Two others are about a wait the runner logs only once per blocker, and they need a second job in the queue while the first one waits: a test with one job can't show them.

Ten seconds of waiting#

The test job waited ten seconds, from its gpu-wait at 16:57:34 to its start at 16:57:44. The proxy's four seconds of noticing cost it nothing: the runner looks every ten seconds, and the engine was gone six seconds before the next look. Before v0.3.1 the same job would have waited for the proxy's idle stop, fifteen minutes after the last request, or for good under steady traffic.

Once a job has logged a gpu-wait, one tick is the shortest wait the runner allows. That is what the first engine the proxy ever started on the A9 cost its benchmark.

Filed under: local-llm, go, proxy, gpu, benchmarks

Read next#

• The benchmark said it was waiting. Once. — where the A9's benchmark hid from the proxy, and the four commits that found it.
• Cutting a stream doesn't cancel the turn — the proxy's engine switching, and the day it sent the A9 a stop for an engine it never started.
• Every way my GPU job queue went quiet — the runner whose gpu-wait line this whole hand-back reads.
• To a coding agent, 503 means try again — why every refusal here is a 423.

Sources#

• systemd.kill(5) — a unit's stop: KillSignal= (SIGTERM) first, then SIGKILL to processes the normal shutdown left behind.
• llama.cpp server README, router mode — the router process forwards each request to a model instance it starts, which inherits its arguments and environment.