Every way my GPU job queue went quiet

· 14 min read
Contents

Six days, one GPU, and every way the queue went quiet: idle with work waiting, stuck with nothing logged, or failing in words that pointed somewhere else.

  • A model download on the net lane: GPU idle 3 h 56 min, 3 h 32 min of it behind one 56.3 GB file at about 3 MB/s.
  • A llama-server left behind by the previous job: GPU lane blocked 3 h 44 min. One event, then nothing.
  • A REQUIRES naming a file nothing would ever write: job 84c could not run for 1 h 07 min and logged nothing. The GPU kept busy; the job just disappeared.
  • One model's launch flags breaking every model's smoke test: the bug lived 16 h 56 min, and three intakes failed their gate, each one looking like the model's fault.
  • A seed script that stops at the first bad job: every job after it is never written. Caught once by a scratch render and reproduced below; no count of what it would have dropped.
  • The watcher that reads the queue: froze on a hung ssh overnight. Not measured, by construction.

TL;DR — A queue that logs state changes cannot report a state that doesn't change. Give every kind of waiting an event or a watcher condition: a download that runs too long, a GPU held by a process you didn't start, a REQUIRES that has been false for hours, and nothing running at all. Make jobs kill their own servers. Render the seed before it reaches the box. Put timeouts on the thing that watches.

Builds on The download that contaminated the benchmark, which covers why the queue has lanes and how it survived its first power cut.

The queue in one screen#

The box is the A9 Mega, a 128 GB Strix Halo mini-PC that runs a backlog of model downloads and agent benchmarks for days at a time. The queue is a directory, ~/jobs, with queue/, running/, done/, failed/, logs/, a state.json of attempt counters and an append-only events.jsonl. A job is a bash file whose header comments are the whole scheduling language. This one was queued on the A9 as I wrote this:

# LANE=net
# REQUIRES=test -f /home/user/work/a9-agent-bench/agent-playground/bench/a9-results/matrix/.done-s2-fnmtpcx-flashnext-iq4xs
# DONE_IF=test -f /home/user/models/MAI-Code-1.1-Flash-On-Device/.ok
# RETRIES=3
bash /home/user/work/a9-agent-bench/agent-playground/bench/fetch-azureml.sh azureml://registries/azureml/models/MAI-Code-1.1-Flash-On-Device/versions/1 /home/user/models/MAI-Code-1.1-Flash-On-Device

runner.py is 203 lines of Python. Every ten seconds it walks the lanes, and a lane picks the first job in name order whose DONE_IF is false and whose REQUIRES is true. The net and gpu lanes are exclusive, because a download once bent a benchmark row. A job is done only when it exits 0 and its DONE_IF agrees. On start, anything left in running/ goes back to queue/ as recovered; that happened nine times in six days, power cuts included. The unit:

[Unit]
Description=A9 job queue (downloads + benchmarks, ~user/jobs)
Wants=network-online.target
After=network-online.target

[Service]
User=user
Environment=HOME=/home/user
ExecStart=/usr/bin/python3 -u /home/user/work/a9-agent-bench/agent-playground/bench/a9-jobs/runner.py --root /home/user/jobs --tick 10
Restart=always
RestartSec=15
KillMode=control-group
TimeoutStopSec=30
LimitMEMLOCK=infinity

[Install]
WantedBy=multi-user.target

LimitMEMLOCK=infinity is there because the NPU runtime needs it, and a job inherits the runner's limits, not your shell's.

Between 10 October 02:00:46 and 15 October 23:59:27 the runner wrote 2678 events: 171 starts, 128 dones, 12 failures, 21 retries, 9 recoveries, 5 gpu-waits.

Downloads go first, even at 3 MB/s#

The lane loop, verbatim:

for lane in ("net", "gpu", "cpu"):
    if lane in self.procs:
        continue
    if lane in EXCLUSIVE and any(l in self.procs for l in EXCLUSIVE):
        continue  # downloads and tests never overlap
    self.start_next(lane)

Net is checked first on every tick. That was a deliberate rule: pending downloads go first, so tests behind them find their files. With Hugging Face at about 107 MB/s, a download costs minutes, not hours. On the evening of 15 October the queue had a model from the public Azure ML registry in it.

{"ts": "2026-10-15 20:09:33", "job": "92b-net-glm53-reap-q4km.job", "event": "start", "lane": "net", "pid": 99494, "attempt": 1}
{"ts": "2026-10-15 20:25:23", "job": "92b-net-glm53-reap-q4km.job", "event": "done", "rc": 0, "verified": true}
{"ts": "2026-10-15 20:25:23", "job": "92c-net-mai-code.job", "event": "start", "lane": "net", "pid": 99681, "attempt": 1}
{"ts": "2026-10-15 23:57:56", "job": "92c-net-mai-code.job", "event": "failed", "rc": 143, "attempt": 1, "verified": false}
{"ts": "2026-10-15 23:57:56", "job": "88b-gpu-build-vulkan-new.job", "event": "start", "lane": "gpu", "pid": 100894, "attempt": 1}

88b, a llama.cpp build with no REQUIRES line, was pulled onto the A9 at 18:31 and started in the same second the download died. Nothing was wrong by the runner's rules. A job was running in an exclusive lane; the other lane waited. The last two progress lines in the job log (curl's \r updates split onto lines):

 64 52.44G  64 33.95G   0      0  2.72M      0 5:27:57 3:32:21 1:55:36  2.92M
 64 52.44G  64 33.95G   0      0  2.72M      0 5:27:57 3:32:22 1:55:35  2.95MTerminated                 bash /home/user/work/a9-agent-bench/agent-playground/bench/fetch-azureml.sh azureml://registries/azureml/models/MAI-Code-1.1-Flash-On-Device/versions/1 /home/user/models/MAI-Code-1.1-Flash-On-Device

64 % of 52.44G after 3 h 32 min, with 1 h 55 min left by curl's estimate. rc 143 is 128 plus SIGTERM. It was stopped at 23:57:56, between the commit that moved it to the back of the queue (23:57:35) and the A9 pulling that commit (23:57:59).

The fix is a REQUIRES that holds a slow download until a late job has finished. That's the .done-s2-fnmtpcx marker in the job above: the download, its intake and its benchmark arm now sort as 98, 98a and 98b, behind everything else. The watcher got a rule too: a net job older than 60 minutes wakes me up.

An orphan from the job before#

The gpu lane refuses to start while a GPU process it didn't start is running. It walks each llama-server's parent chain looking for one of its own job PIDs:

while p > 1:  # walk up the parent chain: is it inside one of our job process trees?

On 11 October an intake job finished, and its llama-server didn't:

{"ts": "2026-10-11 06:06:26", "job": "65-gpu-intake-z-qwen36-35b-q8kxl.job", "event": "done", "rc": 0, "verified": true}
{"ts": "2026-10-11 06:06:26", "job": "65-gpu-intake-z-step37-flash-iq3xxs.job", "event": "gpu-wait", "pid": 46458, "ppid": "1", "cmd": "/home/user/tools/llamacpp-vulkan-master/llama-server --chat-template-file /home/user/work/a9-agent-bench/launch/templates/qwen36-35b.jinja --host 127.0.0.1 --jinja --log-colors off --port 45541 --sleep-"}

Same second. The server was serving qwen36-35b.jinja, the template of the job that had just reported done. Its bash had exited, the server was reparented to PID 1, and from that moment it looked exactly like a stranger. The runner did what it was built to do: logged one gpu-wait naming it, and waited. The next gpu job started at 09:50:20.

This one did produce an event. The gpu-wait event exists because a 2 h 16 min wait in the first days logged nothing, and once it existed nobody read it either. An event is half the fix; the other half is something that wakes up on it.

Intake jobs now end with pkill -x llama-server; sleep 5; pkill -9 -x llama-server, because a wedged server can ignore SIGTERM.

A REQUIRES that can never hold says nothing#

REQUIRES false means "not yet", and the runner skips the job silently, because most of the time "not yet" is correct: the model is still downloading. Here is the template that made job 84c before the fix:

stcheck() { # name config-tag mode
  cat > "$Q/$1.job" <<J
# LANE=gpu
# REQUIRES=test -f $ST/strata-$2.json
# DONE_IF=test -s $RES/measure-strata-$2-$3.jsonl
bash $SE/serve-check.sh $2 $3; rc=\$?; bash $BENCH/a9-jobs/publish.sh; exit \$rc
J
}

84c's tag was qwen-q2_0. The setup job before it wrote strata-q2_0.json, without the family prefix. So 84c waited on strata-qwen-q2_0.json, a file nothing was ever going to write. From 11:58:55, when the setup job gave up, to the fix commit at 13:05:30, it could not run and logged nothing. The setup job, 84b, exited 0 three times and still landed in failed/, because its own DONE_IF looked for the same wrong name. Then 84d, which sorts after 84c, started at 11:58:55 and failed twice in 20 seconds, its log saying no /home/user/tools/strata/strata-qwen-q2_0.json both times. From the outside, a job waiting for a download and a job waiting forever are the same: no line in the log.

The runner still has no event for this. What I'd add, tested against a copy of the runner and not deployed:

@@ -36,6 +36,7 @@
 HDR = re.compile(r"^#\s*(LANE|DONE_IF|REQUIRES|RETRIES)=(.*)$")
 GPU_PROCS = ("llama-server", "llama-bench", "llama-cli", "flm")
 EXCLUSIVE = ("net", "gpu")
+REQUIRES_WARN_S = float(os.environ.get("REQUIRES_WARN_S", 6 * 3600))
 
 
 def parse(path):
@@ -86,6 +87,7 @@
         except (OSError, ValueError):
             self.state = {}
         self.procs = {}  # lane -> (Popen, name)
+        self.blocked = {}  # job -> first time its REQUIRES was seen false
 
     def event(self, job, ev, **kw):
         rec = {"ts": time.strftime("%F %T"), "job": job, "event": ev, **kw}
@@ -142,6 +144,11 @@
                 self.event(name, "skipped", note="DONE_IF already true")
                 continue
             if check(meta["REQUIRES"]) is False:
+                since = self.blocked.setdefault(name, time.time())
+                if since and time.time() - since > REQUIRES_WARN_S:
+                    self.event(name, "requires-wait", requires=meta["REQUIRES"],
+                               waited_s=int(time.time() - since))
+                    self.blocked[name] = 0  # once per runner start
                 continue
             if lane == "gpu":
                 fp = foreign_gpu_user(own)

With a job whose REQUIRES can't hold, REQUIRES_WARN_S=2 and --tick 1:

{"ts": "2026-10-16 12:45:20", "job": "-", "event": "runner-start"}
{"ts": "2026-10-16 12:45:22", "job": "84c-gpu-demo.job", "event": "requires-wait", "requires": "test -f /nonexistent/strata-qwen-q2_0.json", "waited_s": 2}

The timer lives in memory, so it restarts with the runner. Six hours of "not yet" is worth one line either way.

A failure that blamed the model#

New models go through an intake: load at full context, a template check, then a smoke test where three coding clients (opencode, pi, Claude Code) each have to write one file. If the opencode smoke doesn't write it, the gate refuses to grade the model. Between 00:24 and 01:15 on 16 October, three intakes in a row failed that gate. Every smoke log said the same thing:

== smoke / opencode / kimi-linear-q8 
unsupported sampler flag: -b

At 18:41 the day before, MAI-Code's entry in the model roster got SERVE_ARGS="-b 2048 -ub 2048", copied from its shipped config. The script that builds the llama.cpp router presets for every model accepted exactly two short flags, -ctk and -ctv. The fix commit puts the damage plainly: every intake smoke since then exited 1 in 0 s. The job itself exited 0, its DONE_IF found no gate file, and the runner filed it as a failed intake, which is a verdict about a model. Three smoke directories carry that line: two Ling-3.0 builds and Kimi-Linear. The fix landed at 11:37:55 on 16 October.

The queue was loud the whole time. failed events, retries, gate messages. It was just loud in the vocabulary of model failures. Two different models failing the same gate in a row is the tell. I now read the smoke's own run.log before believing a gate.

Seeding: idempotent, until it isn't#

seed.sh writes every job in the plan, about 48 KB of bash, and is rerun whenever the plan changes. Finished jobs skip themselves via DONE_IF. A failed one didn't, at first. Each re-seed put 65-gpu-intake-z-qwen3-coder-30b-q8 back in queue/ while state.json kept counting:

{"ts": "2026-10-14 11:56:25", "job": "65-gpu-intake-z-qwen3-coder-30b-q8.job", "event": "start", "lane": "gpu", "pid": 216852, "attempt": 6}
{"ts": "2026-10-14 11:57:05", "job": "65-gpu-intake-z-qwen3-coder-30b-q8.job", "event": "failed", "rc": -15, "attempt": 6, "verified": false}

Attempt 6, three days after attempt 1. The seed now drops anything already in failed/:

F=${Q%/queue}/failed
for f in "$Q"/*.job; do [ -e "$F/$(basename "$f")" ] && rm -f "$f"; done

Retiring a model needs the same treatment from the other side: its download job goes on an rm list at the end of the seed, or the next re-seed downloads the deleted weights again.

The worse seeding failure is quieter. Jobs are written with unquoted heredocs, so $x expands at seed time and runtime variables need \$x. The bash manual says such a body is "treated similarly to a double-quoted string". Forget one backslash under set -u and the seed aborts at that line. A reduced copy of the pattern, three jobs, the middle one missing its escapes:

job 10-gpu-first  "echo first"
cat > "$Q/20-gpu-typo.job" <<J
# LANE=gpu
echo typo; rc=$?; exit $rc
J
job 30-gpu-third  "echo third"

The seed prints mini-seed.sh: line 11: rc: unbound variable and exits 1. Job 30 is never written. Job 20 is: the redirection created the file before the expansion failed, so it sits in queue/ at 0 bytes. Fed to the real runner:

{"ts": "2026-10-16 12:45:57", "job": "-", "event": "runner-start"}
{"ts": "2026-10-16 12:45:57", "job": "10-gpu-first.job", "event": "start", "lane": "gpu", "pid": 85022, "attempt": 1}
{"ts": "2026-10-16 12:45:57", "job": "20-gpu-typo.job", "event": "start", "lane": "cpu", "pid": 85023, "attempt": 1}
{"ts": "2026-10-16 12:45:58", "job": "10-gpu-first.job", "event": "done", "rc": 0, "verified": null}
{"ts": "2026-10-16 12:45:58", "job": "20-gpu-typo.job", "event": "done", "rc": 0, "verified": null}

An empty file has no LANE, so it runs in the default cpu lane, exits 0, has no DONE_IF to disagree, and lands in done/. That is the reduced repro, not a recorded incident. The one real case was caught by a scratch render before it reached the A9, and nobody counted what it would have dropped. The guard is a render into a scratch HOME before anything is pushed, then bash -n on each job, and the render's exit code is the part that catches it: bash -n passes an empty file.

The watcher, and the night it went quiet#

Events are only useful to something that reads them. This runs from the workstation, one ssh poll every two minutes:

# Wake on: any done/failed/retry event (both lanes), a net job running > 60 min, or nothing running for > 10 min.
start=$(timeout 60 ssh -o ConnectTimeout=10 -o ServerAliveInterval=15 -o ServerAliveCountMax=2 a9 'wc -l < ~/jobs/events.jsonl'); idle=0
while sleep 120; do
  out=$(timeout 90 ssh -o ConnectTimeout=15 -o ServerAliveInterval=15 -o ServerAliveCountMax=2 a9 "tail -n +$((start+1)) ~/jobs/events.jsonl | grep -E '\"event\": \"(done|failed|retry|gpu-wait)\"' | grep -v skipped | head -3; echo ---; ls ~/jobs/running; for f in ~/jobs/running/*-net-*.job; do [ -f \"\$f\" ] && echo NETAGE \$(( (\$(date +%s) - \$(stat -c %Z \"\$f\")) / 60 )); done" 2>/dev/null) || continue
  ev=${out%%---*}; rest=${out#*---}
  [ -n "$(echo "$ev" | tr -d '[:space:]')" ] && { echo "EVENT: $ev"; echo "$rest"; exit 0; }
  age=$(echo "$rest" | sed -n 's/NETAGE //p' | head -1); [ -n "$age" ] && [ "$age" -gt 60 ] && { echo "STALL: net job running ${age} min"; echo "$rest"; exit 0; }
  if [ -z "$(echo "$rest" | grep '\.job')" ]; then idle=$((idle+2)); [ $idle -ge 10 ] && { echo "IDLE: nothing running for ${idle} min"; exit 0; }; else idle=0; fi
done

It exits, which wakes me, on any done, failed, retry or gpu-wait event, a net job running over 60 minutes, or nothing running for 10 minutes. Three of the six silences above were meant to trip it: the orphan and the gate failures within one poll, the download after an hour. As written, it could catch only the download. The last section says why.

The first copy had no timeouts. One night an ssh hung, and the watcher waited on it overnight: the thing watching for silence went silent. Now every call is wrapped in timeout 60/timeout 90, and ServerAliveInterval=15 with ServerAliveCountMax=2 drops a dead connection in about 30 seconds. The ssh_config man page default for ServerAliveInterval is 0: no keepalives at all.

Two silences were still built in, and I found the second one while this post was in review. First, a poll that fails hits || continue, which skips the idle counter too. If the A9 drops off the network, the watcher keeps looping without a word. I added a counter: five failed polls in a row and it exits with UNREACHABLE. Ten minutes after the restart it did exactly that, on a box that was up and running a job.

That was the second silence. The remote command ends in a for loop whose last test, [ -f "$f" ], is false whenever no download is running. A false test as the last command makes ssh exit 1, so the poll counted as failed. With no net job in the queue, which is most of the time, every poll took the || continue branch and nothing else was looked at. This watcher could only ever wake while a download was running. The fix is one word, ; true at the end of the remote command:

$ ssh a9 'for f in ~/jobs/running/*-net-*.job; do [ -f "$f" ] && echo x; done'; echo "old rc=$?"
old rc=1
$ ssh a9 'for f in ~/jobs/running/*-net-*.job; do [ -f "$f" ] && echo x; done; true'; echo "new rc=$?"
new rc=0

The || continue was meant for a flaky network. It also hid an exit code that was wrong on every poll.

Where this queue doesn't fit#

Anything that needs a reboot per side, like an IOMMU or kernel-parameter A/B, can't be a queue job: the runner can't reboot the box it runs on. Those run from the workstation with the queue parked. Everything else, downloads, builds, speed screens, agent benchmark arms, has fit in # LANE= and three more comment lines.

Filed under: linux, systemd, bash, benchmarks, local-llm

Sources#