systemd
- Three shutdowns, three causes
A storage box behind a DJI power station went down three times in two days with the battery near 95 %. The journals gave three different endings.
- The poller that wasn't stuck
One timestamp an hour older than its process sent me after a hung Go poller. A goroutine dump killed that theory; a local-time RTC after DST explained it.
- The config said 32k. The engine still ran 64k.
Adding SIGHUP reload to a Go LLM proxy: a whitelist of three settings, a model list that runs ahead of the engines, and a refresh that finished last.
- The shutdown grace that never ran
A Go proxy's 5-second shutdown grace was dead code. A real drain and socket activation fixed restarts; KillMode, a WaitGroup and a lock needed fixes too.
- Restart a Go server under systemd without dropping requests
A systemd socket holds the port, the Go server drains on SIGTERM, KillMode=mixed spares its children: restarts refuse nothing, cut no stream within the drain.
- Cutting a stream doesn't cancel the turn
A LAN proxy for two GPU boxes learned to switch llama.cpp engines. Cutting streams to switch faster failed: agent clients re-send the turn.
- A text search OOM-killed the whole tmux pane
Eight lines of Python searched a scratch dir, met a 2.7 GB model and needed about 23 GB to read it as text. systemd then stopped the rest of the pane.
- "Terminating immediately" took 90 seconds
systemd and a wrapper's EXIT trap sent llama-server two SIGTERMs. Its Ctrl+C-twice exit() ran inside the signal handler and deadlocked in glibc.
- Every way my GPU job queue went quiet
A file-based job queue ran one GPU box unattended for six days. It never crashed. Here are six ways it went quiet, or nearly did, and the fixes.
- The download that contaminated the benchmark
A model download bent one row of a long-context benchmark. The queue built to stop that got deleted by systemd at boot, then met a real power cut.