Systemd on tsvinc

Three shutdowns, three causes

Mon, 02 Nov 2026 00:00:00 +0000

My storage box sits on a DJI Power 2000 used as a UPS. Over two days, 1 and 2 November, it went down a few times. Some of those had an obvious reason and some had none, and every time I looked, the DJI showed about 95 %. I went in looking for one fault: something that turns a full battery into a dead machine.

There were three, and only one of them was clearly the station.

The config said 32k. The engine still ran 64k.

Thu, 29 Oct 2026 00:00:00 +0000

TL;DR — My Go proxy for local LLMs now reloads three settings on SIGHUP and refuses any other change by name. A reload changes what the proxy tells clients at once, but a running model server keeps the preset it was started with, so the two can disagree until the server restarts; the reload log line says so. Overlapping model-list refreshes are numbered, and only a successful read moves the number forward.

The poller that wasn't stuck

Thu, 29 Oct 2026 00:00:00 +0000

alertd is a Go daemon I run on a Fedora desktop: it polls about 27 public sources of Ukrainian air-raid alert data, one goroutine per source, and stores what they return in SQLite. About a day into a run, one of those pollers looked stuck, and I reached for systemctl --user kill -s QUIT alertd.

It is my favourite kind of command: evidence and a fix in one line. SIGQUIT makes the Go runtime print the stack of every goroutine to stderr, which for a systemd service lands in the journal, and then exit with status 2. Restart=always brings the service straight back up. I had a diagnosis to go with it, too. The poller was stuck, and I knew roughly why. Past me was very sure of that "roughly"; he usually is.

Restart a Go server under systemd without dropping requests

Sun, 25 Oct 2026 00:00:00 +0000

A .socket unit keeps the port open across restarts, the Go server takes that socket from systemd and drains running requests on SIGTERM, and KillMode=mixed leaves the processes those requests wait on alone, so systemctl --user restart refuses no connection and cuts no stream that ends within the drain budget (2 min here).

Tested on: GEEKOM AE8 mini PC, Fedora Linux 44 (kernel 7.2.10), systemd 259 (259.9-1.fc44) user manager with linger on, Go 1.26.0, curl 8.18.0, iproute2 6.17.0 (2026-10-25). The demo service is tsvdemo-drain on 127.0.0.1:18490 (18491 after step 10).
Not verified: system (root) units; IPv6 and Unix-socket ListenStream= addresses (more than one ListenStream= the server refuses, Failure modes); Accept=yes; TLS and HTTP/2; WebSockets and other hijacked connections; a start at boot from the enabled units (I used the runtime unit directory and enable --runtime, no reboot); FreeBind= for a LAN address.
Needs: a working systemctl --user; Go (1.26.0 here); curl; ss; ports 18490 and 18491 free on 127.0.0.1.
Time: 20 min.
Risk: low for the demo: two units on 127.0.0.1 in a unit directory that a reboot empties. On your own service, a change of the socket unit refuses or resets new connections for as long as the drain runs (step 10), and a TimeoutStopSec= below the drain aborts the stop. Every change here is undone in Undo / rollback.

Before you start#

Versions and linger:

$ systemctl --version | head -1; go version; loginctl show-user "$USER" -p Linger
systemd 259 (259.9-1.fc44)
go version go1.26.0 linux/amd64
Linger=yes

The user manager's default stop timeout. It is 45 s here; the systemd 259 man page says 90 s. Fedora lowered it to 45 s in Fedora 38 (the "Shorter Shutdown Timer" change), and /usr/lib/systemd/user.conf lists #DefaultTimeoutStopSec=45s. A drain longer than this default needs its own TimeoutStopSec= (step 4):
```
$ systemctl --user show -p DefaultTimeoutStopUSec
DefaultTimeoutStopUSec=45s
```

Nothing listens on the demo ports, 18490 and 18491 (header line only):

$ ss -ltn '( sport = :18490 or sport = :18491 )'
State Recv-Q Send-Q Local Address:Port Peer Address:Port

Make a directory for the demo and run every command below from it. UNITDIR is where install-units.sh (step 5) puts the units. I used the runtime unit directory, which systemd reads like ~/.config/systemd/user and which lives on tmpfs (/run/user/<UID>), so a reboot removes the trial. For a permanent install leave UNITDIR unset: the script then writes to ~/.config/systemd/user. Export it again in every new shell: without it install-units.sh installs to ~/.config/systemd/user, and the drop-in and Undo commands below lose their directory.
```
mkdir tsvdemo-drain && cd tsvdemo-drain
export UNITDIR=$XDG_RUNTIME_DIR/systemd/user
```
To apply this to your own service later, copy its unit file first (cp ~/.config/systemd/user/<APP>.service ~/<APP>.service.orig, where <APP> is your unit's name). The demo itself replaces nothing.

Steps#

Write the server. Save it as main.go. listenerFrom takes the socket systemd passes (fd 3, LISTEN_PID = this process, LISTEN_FDS = 1) or binds -listen itself; main unsets the four LISTEN_* variables before any child process starts; the first SIGTERM or SIGINT starts the drain and a second one exits at once; serveUntil waits for Shutdown and calls Close when the drain budget runs out; /stream?n=N sends N events one second apart, and &wait=S first runs sleep S as a child process.

// tsvdemo-drain: an SSE demo that takes its listening socket from systemd and drains on stop.
package main

import (
	"context"
	"errors"
	"flag"
	"fmt"
	"log"
	"net"
	"net/http"
	"os"
	"os/exec"
	"os/signal"
	"strconv"
	"syscall"
	"time"
)

func main() {
	listen := flag.String("listen", "127.0.0.1:18490", "address to bind when not socket-activated")
	drain := flag.Duration("drain", 2*time.Minute, "on stop, how long running requests may finish")
	flag.Parse()

	// The listener first, and the LISTEN_* variables gone, before any child process starts.
	ln, inherited, err := listenerFrom(*listen)
	if err != nil {
		log.Fatal(err)
	}
	for _, k := range []string{"LISTEN_PID", "LISTEN_FDS", "LISTEN_FDNAMES", "LISTEN_PIDFDID"} {
		os.Unsetenv(k)
	}

	// First SIGTERM/SIGINT: drain. A second one: exit at once.
	ctx, cancel := context.WithCancel(context.Background())
	sigs := make(chan os.Signal, 1)
	signal.Notify(sigs, os.Interrupt, syscall.SIGTERM)
	go func() {
		<-sigs
		cancel()
		<-sigs
		log.Print("shutdown: second signal, exiting now")
		os.Exit(1)
	}()

	mux := http.NewServeMux()
	mux.HandleFunc("GET /stream", stream)
	srv := &http.Server{Handler: mux, ReadHeaderTimeout: 10 * time.Second}
	log.Printf("pid %d serving on %s (inherited=%t)", os.Getpid(), ln.Addr(), inherited)
	if err := serveUntil(ctx, srv, ln, *drain); err != nil {
		log.Fatal(err)
	}
	log.Print("exit")
}

// listenerFrom returns the socket systemd passed (fd 3, LISTEN_PID = this process,
// LISTEN_FDS = 1) or, without socket activation, binds addr itself.
func listenerFrom(addr string) (net.Listener, bool, error) {
	if os.Getenv("LISTEN_PID") != strconv.Itoa(os.Getpid()) {
		ln, err := net.Listen("tcp", addr)
		return ln, false, err
	}
	if n := os.Getenv("LISTEN_FDS"); n != "1" {
		return nil, false, fmt.Errorf("systemd passed %q sockets, want 1 (one ListenStream= line)", n)
	}
	f := os.NewFile(3, "systemd-socket")
	ln, err := net.FileListener(f) // a dup; close fd 3 itself so children don't inherit it
	f.Close()
	return ln, true, err
}

// serveUntil serves on ln until ctx ends, lets running requests finish for up to drain, then
// closes what is left. It returns only after that: Serve returns ErrServerClosed as soon as
// Shutdown starts, and a main that returned then would cut every running request.
func serveUntil(ctx context.Context, srv *http.Server, ln net.Listener, drain time.Duration) error {
	done := make(chan struct{})
	go func() {
		defer close(done)
		<-ctx.Done()
		log.Printf("shutdown: draining for up to %s", drain)
		sctx, stop := context.WithTimeout(context.Background(), drain)
		defer stop()
		if err := srv.Shutdown(sctx); err != nil {
			log.Printf("shutdown: drain time over (%v), closing", err)
			srv.Close()
			return
		}
		log.Print("shutdown: all requests finished")
	}()
	err := srv.Serve(ln)
	if errors.Is(err, http.ErrServerClosed) {
		<-done
		return nil
	}
	return err
}

// stream sends n events, one per second. With wait=S it first runs `sleep S` as a child
// process: a stand-in for an ssh call or a helper the request waits on.
func stream(w http.ResponseWriter, r *http.Request) {
	n, _ := strconv.Atoi(r.URL.Query().Get("n"))
	if s := r.URL.Query().Get("wait"); s != "" {
		if err := exec.CommandContext(r.Context(), "sleep", s).Run(); err != nil {
			log.Printf("stream: child failed: %v", err)
			http.Error(w, "child failed: "+err.Error(), http.StatusBadGateway)
			return
		}
	}
	w.Header().Set("Content-Type", "text/event-stream")
	rc := http.NewResponseController(w)
	for i := 1; i <= n; i++ {
		fmt.Fprintf(w, "data: pid=%d event=%d/%d\n\n", os.Getpid(), i, n)
		if err := rc.Flush(); err != nil {
			log.Printf("stream: client gone after %d/%d events", i, n)
			return
		}
		if i < n {
			select {
			case <-r.Context().Done():
				log.Printf("stream: cut after %d/%d events", i, n)
				return
			case <-time.After(time.Second):
			}
		}
	}
	log.Printf("stream: %d events sent", n)
}

Create the module and build it:

$ go mod init tsvdemo-drain
go: creating new go.mod: module tsvdemo-drain
go: to add module requirements and sums:
	go mod tidy
$ go vet ./... && go build -o tsvdemo-drain . && ./tsvdemo-drain -h
Usage of ./tsvdemo-drain:
  -drain duration
    	on stop, how long running requests may finish (default 2m0s)
  -listen string
    	address to bind when not socket-activated (default "127.0.0.1:18490")

Expected: no output from go vet, then the two flags.

Check the fallback without systemd. Started by hand, the server binds the address itself; SIGINT drains and exits 0:

$ ./tsvdemo-drain & sleep 0.5; curl -sN 'http://127.0.0.1:18490/stream?n=3'; kill -INT $!; wait $!; echo "exit status $?"
2026/10/25 14:20:56 pid 2187190 serving on 127.0.0.1:18490 (inherited=false)
data: pid=2187190 event=1/3

data: pid=2187190 event=2/3

data: pid=2187190 event=3/3

2026/10/25 14:20:59 stream: 3 events sent
2026/10/25 14:20:59 shutdown: draining for up to 2m0s
2026/10/25 14:20:59 shutdown: all requests finished
2026/10/25 14:20:59 exit
exit status 0

Expected: inherited=false, three events, shutdown: all requests finished, exit status 0. The log lines go to stderr, so they interleave with curl's output. Pasted into an interactive bash, the same line also prints the job lines [1] <PID> before the first log line and [1]+ Done … ./tsvdemo-drain before exit status 0.

Write the socket unit. Save it as tsvdemo-drain.socket:
```
[Unit]
Description=tsvdemo-drain socket: held across restarts, so connections wait while the old process drains

[Socket]
ListenStream=127.0.0.1:18490

[Install]
WantedBy=sockets.target
```
- One ListenStream= line: the server refuses any LISTEN_FDS other than 1 (Failure modes).
- An IP literal with a port: systemd-analyze refuses Go's :18490 and localhost:18490 forms (Failure modes).
- Accept= stays at its default, no: the listening socket itself goes to one service process.
Write the service unit. Save it as tsvdemo-drain.service; install-units.sh replaces @DEMO@ with the demo directory.
```
[Unit]
Description=tsvdemo-drain: SSE demo that drains running requests on stop
# Wants, not Requires: if the socket cannot bind, the service tries to bind the address
# itself and Restart= retries.
Wants=tsvdemo-drain.socket
After=tsvdemo-drain.socket

[Service]
ExecStart=@DEMO@/tsvdemo-drain -listen 127.0.0.1:18490 -drain 2m
Restart=on-failure
RestartSec=5
# SIGTERM to the main process only: children it waits on keep running during the drain.
# Whatever is left in the cgroup gets SIGKILL after the main process exits.
KillMode=mixed
# Above -drain: when the stop times out, systemd ends the drain.
TimeoutStopSec=150

[Install]
WantedBy=default.target
```
- Wants= and After= the socket: starting the service starts the socket first. With Requires=, a socket that cannot bind fails the service as a dependency and nothing retries; with Wants= the server tries to bind by itself, Restart=on-failure retries every 5 s, and the first retry after the port is free gets the socket again (Failure modes).
- KillMode=mixed: the default, control-group, sends SIGTERM to every process in the unit when the stop begins, so a child process a request waits on dies at once (step 8).
- TimeoutStopSec=150, above -drain 2m: when the stop timeout runs out, systemd ends the drain (Failure modes).
- -listen repeats the socket's address. The server uses it only without the socket; change both together.

Write the install script. Save it as install-units.sh. It checks both units with systemd-analyze before it stops anything. A new or changed socket file means stop of socket and service in one call, then start of both: with the service stopped alone, a client that connects during the drain starts it again through the socket, and the socket then refuses to restart while its service runs (Failure modes). An unchanged socket means stop of the service only, while new connections queue on the open socket, then start of both.

#!/bin/bash
# Install or update the tsvdemo-drain units and (re)start them without refusing connections.
set -euo pipefail
cd "$(dirname "$0")"
UNITDIR=${UNITDIR:-$HOME/.config/systemd/user}
mkdir -p "$UNITDIR" build
sed "s|@DEMO@|$PWD|" tsvdemo-drain.service > build/tsvdemo-drain.service
cp tsvdemo-drain.socket build/
systemd-analyze --user verify build/tsvdemo-drain.socket build/tsvdemo-drain.service
if cmp -s build/tsvdemo-drain.socket "$UNITDIR/tsvdemo-drain.socket"; then moved=; else moved=1; fi
install -m644 build/tsvdemo-drain.socket build/tsvdemo-drain.service "$UNITDIR/"
systemctl --user daemon-reload
if [ -n "$moved" ]; then
  # New or changed socket: stop both in one call, so no client re-activates the service in between.
  echo "socket file changed: stopping socket and service, then starting both"
  systemctl --user stop tsvdemo-drain.socket tsvdemo-drain.service
  systemctl --user start tsvdemo-drain.socket tsvdemo-drain.service
else
  # Same socket: stop the service behind it (new connections queue while it drains), start both.
  echo "socket file unchanged: restarting the service behind the open socket"
  systemctl --user stop tsvdemo-drain.service
  systemctl --user start tsvdemo-drain.socket tsvdemo-drain.service
fi

Make it executable:

$ chmod +x install-units.sh

Install, enable and start the units. For a permanent install, leave out --runtime.

$ echo "UNITDIR=$UNITDIR"; ./install-units.sh
UNITDIR=/run/user/1000/systemd/user
socket file changed: stopping socket and service, then starting both
$ systemctl --user enable --runtime tsvdemo-drain.socket tsvdemo-drain.service
Created symlink '/run/user/1000/systemd/user/sockets.target.wants/tsvdemo-drain.socket' → '/run/user/1000/systemd/user/tsvdemo-drain.socket'.
Created symlink '/run/user/1000/systemd/user/default.target.wants/tsvdemo-drain.service' → '/run/user/1000/systemd/user/tsvdemo-drain.service'.

Check that the service got the socket:

$ systemctl --user is-active tsvdemo-drain.socket tsvdemo-drain.service; ss -ltnp 'sport = :18490'
active
active
State  Recv-Q Send-Q Local Address:Port  Peer Address:PortProcess                                    
LISTEN 0      4096       127.0.0.1:18490      0.0.0.0:*    users:(("tsvdemo-drain",pid=2187490,fd=5))
$ journalctl --user -u tsvdemo-drain.service -g serving -n 1 -o cat; tr "\0" "\n" < /proc/$(systemctl --user show -p MainPID --value tsvdemo-drain.service)/environ | grep ^LISTEN_
2026/10/25 14:21:00 pid 2187490 serving on 127.0.0.1:18490 (inherited=true)
LISTEN_PID=2187490
LISTEN_PIDFDID=14783253
LISTEN_FDS=1
LISTEN_FDNAMES=tsvdemo-drain.socket

Expected: both units active; one LISTEN line on 127.0.0.1:18490; inherited=true in the log. /proc/<PID>/environ shows the environment systemd set at exec, including LISTEN_PIDFDID, which systemd 259 adds; the server's os.Unsetenv doesn't change that file, it changes what children get (step 8). If the log says inherited=false, see Failure modes.

Restart while a stream runs. Save restart-test.sh. Stream A asks for 15 events; the restart starts after 3.5 s; request B is made 2 s into the drain; ss shows the listening socket 1 s later. A second argument replaces the restart command (step 10).

#!/bin/bash
# Restart tsvdemo-drain while stream A (15 events) runs; request B is made during the drain.
# usage: restart-test.sh [PORT [COMMAND...]]; COMMAND defaults to systemctl --user restart.
P=${1:-18490}
U=http://127.0.0.1:$P
shift
[ $# -gt 0 ] || set -- systemctl --user restart tsvdemo-drain.service
ts() { date +%T.%3N; }
stamp() { while IFS= read -r l; do [ -n "$l" ] && echo "$(ts) $1 $l"; done; }
{ curl -sSN "$U/stream?n=15" -w 'http=%{http_code}\n' 2>&1; echo "curl rc=$?"; } | stamp A &
sleep 3.5
echo "$(ts) restart begins"
("$@"; echo "$(ts) restart returned rc=$?") &
sleep 2
echo "$(ts) B starts"
{ curl -sSN "$U/stream?n=1" -w 'http=%{http_code} connect=%{time_connect}s total=%{time_total}s\n' 2>&1; echo "curl rc=$?"; } | stamp B &
sleep 1
ss -Hltnp "sport = :$P" | stamp ss
wait

$ chmod +x restart-test.sh
$ ./restart-test.sh
14:21:00.774 A data: pid=2187490 event=1/15
14:21:01.775 A data: pid=2187490 event=2/15
14:21:02.776 A data: pid=2187490 event=3/15
14:21:03.777 A data: pid=2187490 event=4/15
14:21:04.271 restart begins
14:21:04.778 A data: pid=2187490 event=5/15
14:21:05.778 A data: pid=2187490 event=6/15
14:21:06.274 B starts
14:21:06.778 A data: pid=2187490 event=7/15
14:21:07.317 ss LISTEN 1      4096   127.0.0.1:18490 0.0.0.0:*
14:21:07.780 A data: pid=2187490 event=8/15
14:21:08.780 A data: pid=2187490 event=9/15
14:21:09.781 A data: pid=2187490 event=10/15
14:21:10.781 A data: pid=2187490 event=11/15
14:21:11.783 A data: pid=2187490 event=12/15
14:21:12.784 A data: pid=2187490 event=13/15
14:21:13.785 A data: pid=2187490 event=14/15
14:21:14.785 A data: pid=2187490 event=15/15
14:21:14.787 A http=200
14:21:14.789 A curl rc=0
14:21:15.293 restart returned rc=0
14:21:15.306 B data: pid=2188843 event=1/1
14:21:15.307 B http=200 connect=0.000108s total=9.024878s
14:21:15.307 B curl rc=0

Expected: A gets 15 of 15 events from the old PID and curl rc=0. B's connect completes in 0.11 ms, before the new process exists, and B gets its event from the new PID once the drain ends (9.0 s later here). During the drain the old process has already closed its listener (Shutdown closes the listeners first), yet ss shows the port listening, with Recv-Q at 1 while B waits and no process listed: systemd keeps its own copy of the socket. systemctl restart returns when the drain is over (11.0 s here). The journal for the same restart:

$ journalctl --user -u tsvdemo-drain.service --since -30s -o short-precise
Oct 25 14:21:00.455060 host systemd[1397]: Started tsvdemo-drain.service - tsvdemo-drain: SSE demo that drains running requests on stop.
Oct 25 14:21:00.466518 host tsvdemo-drain[2187490]: 2026/10/25 14:21:00 pid 2187490 serving on 127.0.0.1:18490 (inherited=true)
Oct 25 14:21:04.276149 host tsvdemo-drain[2187490]: 2026/10/25 14:21:04 shutdown: draining for up to 2m0s
Oct 25 14:21:04.276225 host systemd[1397]: Stopping tsvdemo-drain.service - tsvdemo-drain: SSE demo that drains running requests on stop...
Oct 25 14:21:14.784301 host tsvdemo-drain[2187490]: 2026/10/25 14:21:14 stream: 15 events sent
Oct 25 14:21:15.288520 host tsvdemo-drain[2187490]: 2026/10/25 14:21:15 shutdown: all requests finished
Oct 25 14:21:15.288520 host tsvdemo-drain[2187490]: 2026/10/25 14:21:15 exit
Oct 25 14:21:15.289782 host systemd[1397]: Stopped tsvdemo-drain.service - tsvdemo-drain: SSE demo that drains running requests on stop.
Oct 25 14:21:15.291662 host systemd[1397]: Started tsvdemo-drain.service - tsvdemo-drain: SSE demo that drains running requests on stop.
Oct 25 14:21:15.304665 host tsvdemo-drain[2188843]: 2026/10/25 14:21:15 pid 2188843 serving on 127.0.0.1:18490 (inherited=true)
Oct 25 14:21:15.305812 host tsvdemo-drain[2188843]: 2026/10/25 14:21:15 stream: 1 events sent

Restart while a request waits on a child process. Save child-test.sh. Request A runs sleep 10 as a child of the server, then streams 3 events; the restart starts 2 s in. The script also prints the child's file descriptors and how many LISTEN_* variables it inherited.

#!/bin/bash
# Restart tsvdemo-drain while request A waits on a child process (sleep 10) before it streams 3 events.
U=http://127.0.0.1:${1:-18490}
ts() { date +%T.%3N; }
stamp() { while IFS= read -r l; do [ -n "$l" ] && echo "$(ts) $1 $l"; done; }
{ curl -sSN "$U/stream?n=3&wait=10" -w 'http=%{http_code}\n' 2>&1; echo "curl rc=$?"; } | stamp A &
sleep 2
main=$(systemctl --user show -p MainPID --value tsvdemo-drain.service)
child=$(pgrep -P "$main" -x sleep)
echo "$(ts) child: pid $child, fds: $(ls /proc/$child/fd | tr '\n' ' ')LISTEN_ variables: $(tr '\0' '\n' < /proc/$child/environ | grep -c '^LISTEN_')"
echo "$(ts) restart begins"
systemctl --user restart tsvdemo-drain.service
echo "$(ts) restart returned rc=$?"
wait

$ chmod +x child-test.sh
$ ./child-test.sh
14:21:17.873 child: pid 2188987, fds: 0 1 2 LISTEN_ variables: 0
14:21:17.878 restart begins
14:21:25.841 A data: pid=2188843 event=1/3
14:21:26.842 A data: pid=2188843 event=2/3
14:21:27.843 A data: pid=2188843 event=3/3
14:21:27.845 A http=200
14:21:27.846 A curl rc=0
14:21:27.988 restart returned rc=0

Expected: the child has only fds 0, 1 and 2 and no LISTEN_* variable; it keeps running through the drain, A gets 3 of 3 events, and the restart returns after them (10.1 s here).

GotchaThe default KillMode kills the child the request waits on

With KillMode=control-group, the SIGTERM of the stop reaches the child too: the request fails with 502 within 6 ms of the restart, and the drain has nothing left to wait for. The same test with a drop-in, then the drop-in removed:

$ mkdir -p "$UNITDIR/tsvdemo-drain.service.d" && printf "[Service]\nKillMode=control-group\n" > "$UNITDIR/tsvdemo-drain.service.d/killmode.conf" && systemctl --user daemon-reload && systemctl --user show -p KillMode tsvdemo-drain.service
KillMode=control-group
$ ./child-test.sh
14:21:30.796 child: pid 2189923, fds: 0 1 2 LISTEN_ variables: 0
14:21:30.801 restart begins
14:21:30.807 A child failed: signal: terminated
14:21:30.808 A http=502
14:21:30.809 A curl rc=0
14:21:30.811 restart returned rc=0
$ rm "$UNITDIR/tsvdemo-drain.service.d/killmode.conf" && rmdir "$UNITDIR/tsvdemo-drain.service.d" && systemctl --user daemon-reload && systemctl --user show -p KillMode tsvdemo-drain.service
KillMode=mixed

End a drain early (optional). A second SIGTERM to the main process makes the server exit at once; a restart then goes on with the new process. The command is systemctl --user kill --kill-whom=main --signal=SIGTERM tsvdemo-drain.service. Save abort-test.sh to see it during a 60-event stream:

#!/bin/bash
# Restart tsvdemo-drain during a 60-event stream, then end the drain early with a second SIGTERM.
U=http://127.0.0.1:${1:-18490}
ts() { date +%T.%3N; }
stamp() { while IFS= read -r l; do [ -n "$l" ] && echo "$(ts) $1 $l"; done; }
{ curl -sSN "$U/stream?n=60" -w 'http=%{http_code}\n' 2>&1; echo "curl rc=$?"; } | stamp A &
sleep 2.5
echo "$(ts) restart begins"
(systemctl --user restart tsvdemo-drain.service; echo "$(ts) restart returned rc=$?") &
sleep 3
echo "$(ts) second SIGTERM to the main process"
systemctl --user kill --kill-whom=main --signal=SIGTERM tsvdemo-drain.service
wait

$ chmod +x abort-test.sh
$ ./abort-test.sh
14:21:31.564 A data: pid=2190293 event=1/60
14:21:32.565 A data: pid=2190293 event=2/60
14:21:33.566 A data: pid=2190293 event=3/60
14:21:34.059 restart begins
14:21:34.566 A data: pid=2190293 event=4/60
14:21:35.566 A data: pid=2190293 event=5/60
14:21:36.567 A data: pid=2190293 event=6/60
14:21:37.062 second SIGTERM to the main process
14:21:37.069 A curl: (18) transfer closed with outstanding read data remaining
14:21:37.070 A http=200
14:21:37.071 A curl rc=18
14:21:37.072 restart returned rc=0

Expected: A is cut (curl: (18)) 3.0 s into the drain, and the restart returns 10 ms after the second signal. The journal records exit status 1 as Failed with result 'exit-code', then starts the new process on the socket:

$ journalctl --user -u tsvdemo-drain.service --since -15s -o short-precise
[…]
Oct 25 14:21:34.065330 host systemd[1397]: Stopping tsvdemo-drain.service - tsvdemo-drain: SSE demo that drains running requests on stop...
Oct 25 14:21:34.065848 host tsvdemo-drain[2190293]: 2026/10/25 14:21:34 shutdown: draining for up to 2m0s
Oct 25 14:21:37.067622 host systemd[1397]: tsvdemo-drain.service: Sent signal SIGTERM to main process 2190293 (tsvdemo-drain) on client request.
Oct 25 14:21:37.068169 host tsvdemo-drain[2190293]: 2026/10/25 14:21:37 shutdown: second signal, exiting now
Oct 25 14:21:37.068962 host systemd[1397]: tsvdemo-drain.service: Main process exited, code=exited, status=1/FAILURE
Oct 25 14:21:37.069055 host systemd[1397]: tsvdemo-drain.service: Failed with result 'exit-code'.
Oct 25 14:21:37.069645 host systemd[1397]: Stopped tsvdemo-drain.service - tsvdemo-drain: SSE demo that drains running requests on stop.
Oct 25 14:21:37.071006 host systemd[1397]: Started tsvdemo-drain.service - tsvdemo-drain: SSE demo that drains running requests on stop.
Oct 25 14:21:37.088652 host tsvdemo-drain[2190806]: 2026/10/25 14:21:37 pid 2190806 serving on 127.0.0.1:18490 (inherited=true)

Apply later changes with install-units.sh. After an edit to the service file only (here its Description=), the socket file is unchanged: the script stops the service behind the open socket and then starts both, so a request made meanwhile queues as in step 7:

$ sed -i 's/^Description=tsvdemo-drain:/Description=tsvdemo-drain (edited):/' tsvdemo-drain.service && ./restart-test.sh 18490 ./install-units.sh
14:21:37.604 A data: pid=2190806 event=1/15
14:21:38.605 A data: pid=2190806 event=2/15
14:21:39.605 A data: pid=2190806 event=3/15
14:21:40.606 A data: pid=2190806 event=4/15
14:21:41.098 restart begins
socket file unchanged: restarting the service behind the open socket
14:21:41.606 A data: pid=2190806 event=5/15
14:21:42.608 A data: pid=2190806 event=6/15
14:21:43.101 B starts
14:21:43.609 A data: pid=2190806 event=7/15
14:21:44.149 ss LISTEN 1      4096   127.0.0.1:18490 0.0.0.0:*
14:21:44.610 A data: pid=2190806 event=8/15
14:21:45.610 A data: pid=2190806 event=9/15
14:21:46.610 A data: pid=2190806 event=10/15
14:21:47.611 A data: pid=2190806 event=11/15
14:21:48.613 A data: pid=2190806 event=12/15
14:21:49.614 A data: pid=2190806 event=13/15
14:21:50.614 A data: pid=2190806 event=14/15
14:21:51.615 A data: pid=2190806 event=15/15
14:21:51.616 A http=200
14:21:51.617 A curl rc=0
Stopping 'tsvdemo-drain.service', but its triggering units are still active:
tsvdemo-drain.socket
14:21:51.727 restart returned rc=0
14:21:51.737 B data: pid=2192112 event=1/1
14:21:51.738 B http=200 connect=0.000108s total=8.627194s
14:21:51.739 B curl rc=0

Expected: socket file unchanged: …, systemd's warning that the socket is still active (that is the point here), B queued and answered by the new PID.

After an edit to the socket file (here the address, in both files), the script stops socket and service in one call before it starts both:

$ sed -i 's/127.0.0.1:18490/127.0.0.1:18491/' tsvdemo-drain.socket tsvdemo-drain.service && ./install-units.sh
socket file changed: stopping socket and service, then starting both
$ ss -ltnp '( sport = :18490 or sport = :18491 )'; journalctl --user -u tsvdemo-drain.service -g serving -n 1 -o cat
State  Recv-Q Send-Q Local Address:Port  Peer Address:PortProcess                                    
LISTEN 0      4096       127.0.0.1:18491      0.0.0.0:*    users:(("tsvdemo-drain",pid=2192375,fd=5))
2026/10/25 14:21:52 pid 2192375 serving on 127.0.0.1:18491 (inherited=true)

Expected: socket file changed: …, one LISTEN line on 18491, and inherited=true on 18491.

NoteA socket change refuses or resets new connections until the drain ends

In a separate run of the same move (18490 to 18491) with a stream running, two requests made during the drain, one to each port, both failed at once with curl: (7) … Could not connect to server, while the stream on the old port finished 8 of 8. With the address unchanged and another line of the socket file edited, a request made during the drain connected, waited, and got curl: (56) Recv failure: Connection reset by peer when the socket closed after the service. A plain restart (steps 7–9) has neither problem.

The shutdown grace that never ran

Sun, 25 Oct 2026 00:00:00 +0000

The net/http documentation spends two sentences on the trap, in the comment on Server.Shutdown:

// When Shutdown is called, [Serve], [ServeTLS], [ListenAndServe], and
// [ListenAndServeTLS] immediately return [ErrServerClosed]. Make sure the
// program doesn't exit and waits instead for Shutdown to return.

My LAN proxy for local models did the other thing. It started Shutdown in a goroutine with a five-second budget, let ListenAndServe return, and returned from main. Until v0.3.0, every restart cut every stream going through it within milliseconds of the SIGTERM, logged nothing about the requests it cut, and exited with status 0. The proxy needs a restart after every config edit, and its streams are coding agents' turns against models on two GPU boxes, which can run for minutes.

"Terminating immediately" took 90 seconds

Sat, 24 Oct 2026 00:00:00 +0000

llama-server keeps a message for the moment you run out of patience with it: Received second interrupt, terminating immediately. On the 3090 box it printed that at 09:45:04. The next thing systemd had to say about the unit came at 09:46:34, and it was Aborting.

That stop was half of an engine switch. My LAN proxy for local models stops one llama.cpp build and starts another when a request names a model that only the other build has (why it does that). The proxy logged the new engine up after 1m36s. A switch between those two builds takes 2 to 4 seconds. Two signals did this, the second one came from my own script, and the "immediately" path deadlocked inside glibc.

A text search OOM-killed the whole tmux pane

Sat, 24 Oct 2026 00:00:00 +0000

22:57:55: eight lines of Python start searching a scratch directory for one log line.

23:01:49: the kernel kills them. They hold 15.3 GiB of RAM and another 1.4 GiB of swap.

By 23:01:51, systemd has stopped everything else in that tmux pane: the interactive session, a context probe nearly four minutes into a 262k-token request, a go test -count=100.

All of it to find one log line.

The search, and what it walked into#

for root,ds,fs in os.walk('agent-work'):
    if 'review-cutting' in root or 'client-research' in root: continue
    for f in fs:
        p=os.path.join(root,f)
        try: t=open(p,errors='replace').read()
        except: continue
        for l in t.splitlines():
            if 'host a9: engine' in l and 'up after' in l: print(p, l[:200])

It was checking a single fact for a blog post: which log lines said when a GPU box's engine had come up. The command eight seconds before it had filtered by extension, .log .txt .out .err. This one didn't. It walked the whole scratch tree, which held, among the logs, a 2.7 GB GGUF model downloaded that evening for another job.

Cutting a stream doesn't cancel the turn

Sat, 24 Oct 2026 00:00:00 +0000

The plan was a router. One OpenAI- and Anthropic-compatible URL on a mini-PC, two GPU boxes behind it, read the request's model, pick the box, forward the bytes. I started it at 23:37 one night. By 22:04 the next evening it was starting and stopping llama.cpp builds on the 3090 box (the A9's benchmarks kept it out that day), deciding when a request has to wait, and shutting engines down after fifteen idle minutes so a benchmark could have the GPU back.

Every way my GPU job queue went quiet

Fri, 16 Oct 2026 00:00:00 +0000

Six days, one GPU, and every way the queue went quiet: idle with work waiting, stuck with nothing logged, or failing in words that pointed somewhere else.

A model download on the net lane: GPU idle 3 h 56 min, 3 h 32 min of it behind one 56.3 GB file at about 3 MB/s.
A llama-server left behind by the previous job: GPU lane blocked 3 h 44 min. One event, then nothing.
A REQUIRES naming a file nothing would ever write: job 84c could not run for 1 h 07 min and logged nothing. The GPU kept busy; the job just disappeared.
One model's launch flags breaking every model's smoke test: the bug lived 16 h 56 min, and three intakes failed their gate, each one looking like the model's fault.
A seed script that stops at the first bad job: every job after it is never written. Caught once by a scratch render and reproduced below; no count of what it would have dropped.
The watcher that reads the queue: froze on a hung ssh overnight. Not measured, by construction.

TL;DR — A queue that logs state changes cannot report a state that doesn't change. Give every kind of waiting an event or a watcher condition: a download that runs too long, a GPU held by a process you didn't start, a REQUIRES that has been false for hours, and nothing running at all. Make jobs kill their own servers. Render the seed before it reaches the box. Put timeouts on the thing that watches.

The download that contaminated the benchmark

Sat, 10 Oct 2026 00:00:00 +0000

$ git show --name-status --relative --format='%h %ad %s' --date=format:'%H:%M' dfd9f06 -- a9-results
dfd9f06 01:59 a9-results: gpt-oss long-ctx run overlapped downloads, marked contaminated; rerun queued

R100	a9-results/longctx-gptoss120b-b11538.jsonl	a9-results/longctx-gptoss120b-b11538.contaminated-by-download.jsonl

That rename is a confession. The file holds gpt-oss-120b's long-context sweep on the A9 Mega: llama-bench prefill and generation speed at depth 0, 32k, 64k and 96k tokens. It started at 01:18. The model downloads started at 01:19.

R100 means the content didn't change by a byte. The numbers stayed exactly as they were, and the filename now says they can't be trusted. The commit subject says the rest: rerun queued. There was no queue running yet. The runner's first commit had landed 25 seconds before the rename; the service started 94 seconds after it.

Hardening the box you're logged in through

Mon, 20 Jul 2026 00:00:00 +0000

Assumes you've SSH'd into something you couldn't physically reach.

The last thing I did in that thread was switch off the instrument.

pkill -f 'port-test'

A loop had been polling Transmission's port-test every fifteen seconds — an external server asked to connect back to the public IP and report whether it got through. It had been printing false for minutes. I had just read the router's dst-nat table, found a one-digit typo in it, fixed the typo, and then explained at length that the typo didn't matter, because the ISP was doing carrier-grade NAT and no port forward could ever work. Then I killed the poller, on the reasoning that it couldn't flip true from there.

Transmission is a remote-code-execution box by default

Tue, 07 Jul 2026 00:00:00 +0000

TL;DR Transmission's script-torrent-done-filename runs an executable of your choosing as the daemon user every time a torrent completes, and it is settable over the RPC interface — so anything that can reach the RPC port and pass its auth chooses what the daemon executes. That is remote code execution by design, not a bug, and you cannot seccomp it away because the daemon legitimately needs execve. Narrow who can reach the RPC (the whitelist takes shell globs, not CIDR), then cage what an escaped command can touch with a systemd sandbox.

The obvious way to stop a program from running arbitrary commands is to take away its ability to run commands. systemd has a knob for exactly this: