Manual

Restart a Go server under systemd without dropping requests

last verified
Contents

A .socket unit keeps the port open across restarts, the Go server takes that socket from systemd and drains running requests on SIGTERM, and KillMode=mixed leaves the processes those requests wait on alone, so systemctl --user restart refuses no connection and cuts no stream that ends within the drain budget (2 min here).

Tested on
GEEKOM AE8 mini PC, Fedora Linux 44 (kernel 7.2.10), systemd 259 (259.9-1.fc44) user manager with linger on, Go 1.26.0, curl 8.18.0, iproute2 6.17.0 (2026-10-25). The demo service is tsvdemo-drain on 127.0.0.1:18490 (18491 after step 10).
Not verified
system (root) units; IPv6 and Unix-socket ListenStream= addresses (more than one ListenStream= the server refuses, Failure modes); Accept=yes; TLS and HTTP/2; WebSockets and other hijacked connections; a start at boot from the enabled units (I used the runtime unit directory and enable --runtime, no reboot); FreeBind= for a LAN address.
Needs
a working systemctl --user; Go (1.26.0 here); curl; ss; ports 18490 and 18491 free on 127.0.0.1.
Time
20 min.
Risk
low for the demo: two units on 127.0.0.1 in a unit directory that a reboot empties. On your own service, a change of the socket unit refuses or resets new connections for as long as the drain runs (step 10), and a TimeoutStopSec= below the drain aborts the stop. Every change here is undone in Undo / rollback.

Before you start#

  • Versions and linger:
    $ systemctl --version | head -1; go version; loginctl show-user "$USER" -p Linger
systemd 259 (259.9-1.fc44)
go version go1.26.0 linux/amd64
Linger=yes
  • The user manager's default stop timeout. It is 45 s here; the systemd 259 man page says 90 s. Fedora lowered it to 45 s in Fedora 38 (the "Shorter Shutdown Timer" change), and /usr/lib/systemd/user.conf lists #DefaultTimeoutStopSec=45s. A drain longer than this default needs its own TimeoutStopSec= (step 4):
    $ systemctl --user show -p DefaultTimeoutStopUSec
DefaultTimeoutStopUSec=45s
  • Nothing listens on the demo ports, 18490 and 18491 (header line only):
    $ ss -ltn '( sport = :18490 or sport = :18491 )'
State Recv-Q Send-Q Local Address:Port Peer Address:Port
  • Make a directory for the demo and run every command below from it. UNITDIR is where install-units.sh (step 5) puts the units. I used the runtime unit directory, which systemd reads like ~/.config/systemd/user and which lives on tmpfs (/run/user/<UID>), so a reboot removes the trial. For a permanent install leave UNITDIR unset: the script then writes to ~/.config/systemd/user. Export it again in every new shell: without it install-units.sh installs to ~/.config/systemd/user, and the drop-in and Undo commands below lose their directory.
    mkdir tsvdemo-drain && cd tsvdemo-drain
export UNITDIR=$XDG_RUNTIME_DIR/systemd/user
  • To apply this to your own service later, copy its unit file first (cp ~/.config/systemd/user/<APP>.service ~/<APP>.service.orig, where <APP> is your unit's name). The demo itself replaces nothing.

Steps#

  1. Write the server. Save it as main.go. listenerFrom takes the socket systemd passes (fd 3, LISTEN_PID = this process, LISTEN_FDS = 1) or binds -listen itself; main unsets the four LISTEN_* variables before any child process starts; the first SIGTERM or SIGINT starts the drain and a second one exits at once; serveUntil waits for Shutdown and calls Close when the drain budget runs out; /stream?n=N sends N events one second apart, and &wait=S first runs sleep S as a child process.

    // tsvdemo-drain: an SSE demo that takes its listening socket from systemd and drains on stop.
package main

import (
	"context"
	"errors"
	"flag"
	"fmt"
	"log"
	"net"
	"net/http"
	"os"
	"os/exec"
	"os/signal"
	"strconv"
	"syscall"
	"time"
)

func main() {
	listen := flag.String("listen", "127.0.0.1:18490", "address to bind when not socket-activated")
	drain := flag.Duration("drain", 2*time.Minute, "on stop, how long running requests may finish")
	flag.Parse()

	// The listener first, and the LISTEN_* variables gone, before any child process starts.
	ln, inherited, err := listenerFrom(*listen)
	if err != nil {
		log.Fatal(err)
	}
	for _, k := range []string{"LISTEN_PID", "LISTEN_FDS", "LISTEN_FDNAMES", "LISTEN_PIDFDID"} {
		os.Unsetenv(k)
	}

	// First SIGTERM/SIGINT: drain. A second one: exit at once.
	ctx, cancel := context.WithCancel(context.Background())
	sigs := make(chan os.Signal, 1)
	signal.Notify(sigs, os.Interrupt, syscall.SIGTERM)
	go func() {
		<-sigs
		cancel()
		<-sigs
		log.Print("shutdown: second signal, exiting now")
		os.Exit(1)
	}()

	mux := http.NewServeMux()
	mux.HandleFunc("GET /stream", stream)
	srv := &http.Server{Handler: mux, ReadHeaderTimeout: 10 * time.Second}
	log.Printf("pid %d serving on %s (inherited=%t)", os.Getpid(), ln.Addr(), inherited)
	if err := serveUntil(ctx, srv, ln, *drain); err != nil {
		log.Fatal(err)
	}
	log.Print("exit")
}

// listenerFrom returns the socket systemd passed (fd 3, LISTEN_PID = this process,
// LISTEN_FDS = 1) or, without socket activation, binds addr itself.
func listenerFrom(addr string) (net.Listener, bool, error) {
	if os.Getenv("LISTEN_PID") != strconv.Itoa(os.Getpid()) {
		ln, err := net.Listen("tcp", addr)
		return ln, false, err
	}
	if n := os.Getenv("LISTEN_FDS"); n != "1" {
		return nil, false, fmt.Errorf("systemd passed %q sockets, want 1 (one ListenStream= line)", n)
	}
	f := os.NewFile(3, "systemd-socket")
	ln, err := net.FileListener(f) // a dup; close fd 3 itself so children don't inherit it
	f.Close()
	return ln, true, err
}

// serveUntil serves on ln until ctx ends, lets running requests finish for up to drain, then
// closes what is left. It returns only after that: Serve returns ErrServerClosed as soon as
// Shutdown starts, and a main that returned then would cut every running request.
func serveUntil(ctx context.Context, srv *http.Server, ln net.Listener, drain time.Duration) error {
	done := make(chan struct{})
	go func() {
		defer close(done)
		<-ctx.Done()
		log.Printf("shutdown: draining for up to %s", drain)
		sctx, stop := context.WithTimeout(context.Background(), drain)
		defer stop()
		if err := srv.Shutdown(sctx); err != nil {
			log.Printf("shutdown: drain time over (%v), closing", err)
			srv.Close()
			return
		}
		log.Print("shutdown: all requests finished")
	}()
	err := srv.Serve(ln)
	if errors.Is(err, http.ErrServerClosed) {
		<-done
		return nil
	}
	return err
}

// stream sends n events, one per second. With wait=S it first runs `sleep S` as a child
// process: a stand-in for an ssh call or a helper the request waits on.
func stream(w http.ResponseWriter, r *http.Request) {
	n, _ := strconv.Atoi(r.URL.Query().Get("n"))
	if s := r.URL.Query().Get("wait"); s != "" {
		if err := exec.CommandContext(r.Context(), "sleep", s).Run(); err != nil {
			log.Printf("stream: child failed: %v", err)
			http.Error(w, "child failed: "+err.Error(), http.StatusBadGateway)
			return
		}
	}
	w.Header().Set("Content-Type", "text/event-stream")
	rc := http.NewResponseController(w)
	for i := 1; i <= n; i++ {
		fmt.Fprintf(w, "data: pid=%d event=%d/%d\n\n", os.Getpid(), i, n)
		if err := rc.Flush(); err != nil {
			log.Printf("stream: client gone after %d/%d events", i, n)
			return
		}
		if i < n {
			select {
			case <-r.Context().Done():
				log.Printf("stream: cut after %d/%d events", i, n)
				return
			case <-time.After(time.Second):
			}
		}
	}
	log.Printf("stream: %d events sent", n)
}

    Create the module and build it:

    $ go mod init tsvdemo-drain
go: creating new go.mod: module tsvdemo-drain
go: to add module requirements and sums:
	go mod tidy
$ go vet ./... && go build -o tsvdemo-drain . && ./tsvdemo-drain -h
Usage of ./tsvdemo-drain:
  -drain duration
    	on stop, how long running requests may finish (default 2m0s)
  -listen string
    	address to bind when not socket-activated (default "127.0.0.1:18490")

    Expected: no output from go vet, then the two flags.

  2. Check the fallback without systemd. Started by hand, the server binds the address itself; SIGINT drains and exits 0:

    $ ./tsvdemo-drain & sleep 0.5; curl -sN 'http://127.0.0.1:18490/stream?n=3'; kill -INT $!; wait $!; echo "exit status $?"
2026/10/25 14:20:56 pid 2187190 serving on 127.0.0.1:18490 (inherited=false)
data: pid=2187190 event=1/3

data: pid=2187190 event=2/3

data: pid=2187190 event=3/3

2026/10/25 14:20:59 stream: 3 events sent
2026/10/25 14:20:59 shutdown: draining for up to 2m0s
2026/10/25 14:20:59 shutdown: all requests finished
2026/10/25 14:20:59 exit
exit status 0

    Expected: inherited=false, three events, shutdown: all requests finished, exit status 0. The log lines go to stderr, so they interleave with curl's output. Pasted into an interactive bash, the same line also prints the job lines [1] <PID> before the first log line and [1]+ Done … ./tsvdemo-drain before exit status 0.

  3. Write the socket unit. Save it as tsvdemo-drain.socket:

    [Unit]
Description=tsvdemo-drain socket: held across restarts, so connections wait while the old process drains

[Socket]
ListenStream=127.0.0.1:18490

[Install]
WantedBy=sockets.target
    • One ListenStream= line: the server refuses any LISTEN_FDS other than 1 (Failure modes).
    • An IP literal with a port: systemd-analyze refuses Go's :18490 and localhost:18490 forms (Failure modes).
    • Accept= stays at its default, no: the listening socket itself goes to one service process.

  4. Write the service unit. Save it as tsvdemo-drain.service; install-units.sh replaces @DEMO@ with the demo directory.

    [Unit]
Description=tsvdemo-drain: SSE demo that drains running requests on stop
# Wants, not Requires: if the socket cannot bind, the service tries to bind the address
# itself and Restart= retries.
Wants=tsvdemo-drain.socket
After=tsvdemo-drain.socket

[Service]
ExecStart=@DEMO@/tsvdemo-drain -listen 127.0.0.1:18490 -drain 2m
Restart=on-failure
RestartSec=5
# SIGTERM to the main process only: children it waits on keep running during the drain.
# Whatever is left in the cgroup gets SIGKILL after the main process exits.
KillMode=mixed
# Above -drain: when the stop times out, systemd ends the drain.
TimeoutStopSec=150

[Install]
WantedBy=default.target
    • Wants= and After= the socket: starting the service starts the socket first. With Requires=, a socket that cannot bind fails the service as a dependency and nothing retries; with Wants= the server tries to bind by itself, Restart=on-failure retries every 5 s, and the first retry after the port is free gets the socket again (Failure modes).
    • KillMode=mixed: the default, control-group, sends SIGTERM to every process in the unit when the stop begins, so a child process a request waits on dies at once (step 8).
    • TimeoutStopSec=150, above -drain 2m: when the stop timeout runs out, systemd ends the drain (Failure modes).
    • -listen repeats the socket's address. The server uses it only without the socket; change both together.

  5. Write the install script. Save it as install-units.sh. It checks both units with systemd-analyze before it stops anything. A new or changed socket file means stop of socket and service in one call, then start of both: with the service stopped alone, a client that connects during the drain starts it again through the socket, and the socket then refuses to restart while its service runs (Failure modes). An unchanged socket means stop of the service only, while new connections queue on the open socket, then start of both.

    #!/bin/bash
# Install or update the tsvdemo-drain units and (re)start them without refusing connections.
set -euo pipefail
cd "$(dirname "$0")"
UNITDIR=${UNITDIR:-$HOME/.config/systemd/user}
mkdir -p "$UNITDIR" build
sed "s|@DEMO@|$PWD|" tsvdemo-drain.service > build/tsvdemo-drain.service
cp tsvdemo-drain.socket build/
systemd-analyze --user verify build/tsvdemo-drain.socket build/tsvdemo-drain.service
if cmp -s build/tsvdemo-drain.socket "$UNITDIR/tsvdemo-drain.socket"; then moved=; else moved=1; fi
install -m644 build/tsvdemo-drain.socket build/tsvdemo-drain.service "$UNITDIR/"
systemctl --user daemon-reload
if [ -n "$moved" ]; then
  # New or changed socket: stop both in one call, so no client re-activates the service in between.
  echo "socket file changed: stopping socket and service, then starting both"
  systemctl --user stop tsvdemo-drain.socket tsvdemo-drain.service
  systemctl --user start tsvdemo-drain.socket tsvdemo-drain.service
else
  # Same socket: stop the service behind it (new connections queue while it drains), start both.
  echo "socket file unchanged: restarting the service behind the open socket"
  systemctl --user stop tsvdemo-drain.service
  systemctl --user start tsvdemo-drain.socket tsvdemo-drain.service
fi

    Make it executable:

    $ chmod +x install-units.sh

  6. Install, enable and start the units. For a permanent install, leave out --runtime.

    $ echo "UNITDIR=$UNITDIR"; ./install-units.sh
UNITDIR=/run/user/1000/systemd/user
socket file changed: stopping socket and service, then starting both
$ systemctl --user enable --runtime tsvdemo-drain.socket tsvdemo-drain.service
Created symlink '/run/user/1000/systemd/user/sockets.target.wants/tsvdemo-drain.socket' → '/run/user/1000/systemd/user/tsvdemo-drain.socket'.
Created symlink '/run/user/1000/systemd/user/default.target.wants/tsvdemo-drain.service' → '/run/user/1000/systemd/user/tsvdemo-drain.service'.

    Check that the service got the socket:

    $ systemctl --user is-active tsvdemo-drain.socket tsvdemo-drain.service; ss -ltnp 'sport = :18490'
active
active
State  Recv-Q Send-Q Local Address:Port  Peer Address:PortProcess                                    
LISTEN 0      4096       127.0.0.1:18490      0.0.0.0:*    users:(("tsvdemo-drain",pid=2187490,fd=5))
$ journalctl --user -u tsvdemo-drain.service -g serving -n 1 -o cat; tr "\0" "\n" < /proc/$(systemctl --user show -p MainPID --value tsvdemo-drain.service)/environ | grep ^LISTEN_
2026/10/25 14:21:00 pid 2187490 serving on 127.0.0.1:18490 (inherited=true)
LISTEN_PID=2187490
LISTEN_PIDFDID=14783253
LISTEN_FDS=1
LISTEN_FDNAMES=tsvdemo-drain.socket

    Expected: both units active; one LISTEN line on 127.0.0.1:18490; inherited=true in the log. /proc/<PID>/environ shows the environment systemd set at exec, including LISTEN_PIDFDID, which systemd 259 adds; the server's os.Unsetenv doesn't change that file, it changes what children get (step 8). If the log says inherited=false, see Failure modes.

  7. Restart while a stream runs. Save restart-test.sh. Stream A asks for 15 events; the restart starts after 3.5 s; request B is made 2 s into the drain; ss shows the listening socket 1 s later. A second argument replaces the restart command (step 10).

    #!/bin/bash
# Restart tsvdemo-drain while stream A (15 events) runs; request B is made during the drain.
# usage: restart-test.sh [PORT [COMMAND...]]; COMMAND defaults to systemctl --user restart.
P=${1:-18490}
U=http://127.0.0.1:$P
shift
[ $# -gt 0 ] || set -- systemctl --user restart tsvdemo-drain.service
ts() { date +%T.%3N; }
stamp() { while IFS= read -r l; do [ -n "$l" ] && echo "$(ts) $1 $l"; done; }
{ curl -sSN "$U/stream?n=15" -w 'http=%{http_code}\n' 2>&1; echo "curl rc=$?"; } | stamp A &
sleep 3.5
echo "$(ts) restart begins"
("$@"; echo "$(ts) restart returned rc=$?") &
sleep 2
echo "$(ts) B starts"
{ curl -sSN "$U/stream?n=1" -w 'http=%{http_code} connect=%{time_connect}s total=%{time_total}s\n' 2>&1; echo "curl rc=$?"; } | stamp B &
sleep 1
ss -Hltnp "sport = :$P" | stamp ss
wait

    $ chmod +x restart-test.sh
$ ./restart-test.sh
14:21:00.774 A data: pid=2187490 event=1/15
14:21:01.775 A data: pid=2187490 event=2/15
14:21:02.776 A data: pid=2187490 event=3/15
14:21:03.777 A data: pid=2187490 event=4/15
14:21:04.271 restart begins
14:21:04.778 A data: pid=2187490 event=5/15
14:21:05.778 A data: pid=2187490 event=6/15
14:21:06.274 B starts
14:21:06.778 A data: pid=2187490 event=7/15
14:21:07.317 ss LISTEN 1      4096   127.0.0.1:18490 0.0.0.0:*
14:21:07.780 A data: pid=2187490 event=8/15
14:21:08.780 A data: pid=2187490 event=9/15
14:21:09.781 A data: pid=2187490 event=10/15
14:21:10.781 A data: pid=2187490 event=11/15
14:21:11.783 A data: pid=2187490 event=12/15
14:21:12.784 A data: pid=2187490 event=13/15
14:21:13.785 A data: pid=2187490 event=14/15
14:21:14.785 A data: pid=2187490 event=15/15
14:21:14.787 A http=200
14:21:14.789 A curl rc=0
14:21:15.293 restart returned rc=0
14:21:15.306 B data: pid=2188843 event=1/1
14:21:15.307 B http=200 connect=0.000108s total=9.024878s
14:21:15.307 B curl rc=0

    Expected: A gets 15 of 15 events from the old PID and curl rc=0. B's connect completes in 0.11 ms, before the new process exists, and B gets its event from the new PID once the drain ends (9.0 s later here). During the drain the old process has already closed its listener (Shutdown closes the listeners first), yet ss shows the port listening, with Recv-Q at 1 while B waits and no process listed: systemd keeps its own copy of the socket. systemctl restart returns when the drain is over (11.0 s here). The journal for the same restart:

    $ journalctl --user -u tsvdemo-drain.service --since -30s -o short-precise
Oct 25 14:21:00.455060 host systemd[1397]: Started tsvdemo-drain.service - tsvdemo-drain: SSE demo that drains running requests on stop.
Oct 25 14:21:00.466518 host tsvdemo-drain[2187490]: 2026/10/25 14:21:00 pid 2187490 serving on 127.0.0.1:18490 (inherited=true)
Oct 25 14:21:04.276149 host tsvdemo-drain[2187490]: 2026/10/25 14:21:04 shutdown: draining for up to 2m0s
Oct 25 14:21:04.276225 host systemd[1397]: Stopping tsvdemo-drain.service - tsvdemo-drain: SSE demo that drains running requests on stop...
Oct 25 14:21:14.784301 host tsvdemo-drain[2187490]: 2026/10/25 14:21:14 stream: 15 events sent
Oct 25 14:21:15.288520 host tsvdemo-drain[2187490]: 2026/10/25 14:21:15 shutdown: all requests finished
Oct 25 14:21:15.288520 host tsvdemo-drain[2187490]: 2026/10/25 14:21:15 exit
Oct 25 14:21:15.289782 host systemd[1397]: Stopped tsvdemo-drain.service - tsvdemo-drain: SSE demo that drains running requests on stop.
Oct 25 14:21:15.291662 host systemd[1397]: Started tsvdemo-drain.service - tsvdemo-drain: SSE demo that drains running requests on stop.
Oct 25 14:21:15.304665 host tsvdemo-drain[2188843]: 2026/10/25 14:21:15 pid 2188843 serving on 127.0.0.1:18490 (inherited=true)
Oct 25 14:21:15.305812 host tsvdemo-drain[2188843]: 2026/10/25 14:21:15 stream: 1 events sent

  8. Restart while a request waits on a child process. Save child-test.sh. Request A runs sleep 10 as a child of the server, then streams 3 events; the restart starts 2 s in. The script also prints the child's file descriptors and how many LISTEN_* variables it inherited.

    #!/bin/bash
# Restart tsvdemo-drain while request A waits on a child process (sleep 10) before it streams 3 events.
U=http://127.0.0.1:${1:-18490}
ts() { date +%T.%3N; }
stamp() { while IFS= read -r l; do [ -n "$l" ] && echo "$(ts) $1 $l"; done; }
{ curl -sSN "$U/stream?n=3&wait=10" -w 'http=%{http_code}\n' 2>&1; echo "curl rc=$?"; } | stamp A &
sleep 2
main=$(systemctl --user show -p MainPID --value tsvdemo-drain.service)
child=$(pgrep -P "$main" -x sleep)
echo "$(ts) child: pid $child, fds: $(ls /proc/$child/fd | tr '\n' ' ')LISTEN_ variables: $(tr '\0' '\n' < /proc/$child/environ | grep -c '^LISTEN_')"
echo "$(ts) restart begins"
systemctl --user restart tsvdemo-drain.service
echo "$(ts) restart returned rc=$?"
wait

    $ chmod +x child-test.sh
$ ./child-test.sh
14:21:17.873 child: pid 2188987, fds: 0 1 2 LISTEN_ variables: 0
14:21:17.878 restart begins
14:21:25.841 A data: pid=2188843 event=1/3
14:21:26.842 A data: pid=2188843 event=2/3
14:21:27.843 A data: pid=2188843 event=3/3
14:21:27.845 A http=200
14:21:27.846 A curl rc=0
14:21:27.988 restart returned rc=0

    Expected: the child has only fds 0, 1 and 2 and no LISTEN_* variable; it keeps running through the drain, A gets 3 of 3 events, and the restart returns after them (10.1 s here).

    GotchaThe default KillMode kills the child the request waits on

    With KillMode=control-group, the SIGTERM of the stop reaches the child too: the request fails with 502 within 6 ms of the restart, and the drain has nothing left to wait for. The same test with a drop-in, then the drop-in removed:

    $ mkdir -p "$UNITDIR/tsvdemo-drain.service.d" && printf "[Service]\nKillMode=control-group\n" > "$UNITDIR/tsvdemo-drain.service.d/killmode.conf" && systemctl --user daemon-reload && systemctl --user show -p KillMode tsvdemo-drain.service
KillMode=control-group
$ ./child-test.sh
14:21:30.796 child: pid 2189923, fds: 0 1 2 LISTEN_ variables: 0
14:21:30.801 restart begins
14:21:30.807 A child failed: signal: terminated
14:21:30.808 A http=502
14:21:30.809 A curl rc=0
14:21:30.811 restart returned rc=0
$ rm "$UNITDIR/tsvdemo-drain.service.d/killmode.conf" && rmdir "$UNITDIR/tsvdemo-drain.service.d" && systemctl --user daemon-reload && systemctl --user show -p KillMode tsvdemo-drain.service
KillMode=mixed

  9. End a drain early (optional). A second SIGTERM to the main process makes the server exit at once; a restart then goes on with the new process. The command is systemctl --user kill --kill-whom=main --signal=SIGTERM tsvdemo-drain.service. Save abort-test.sh to see it during a 60-event stream:

    #!/bin/bash
# Restart tsvdemo-drain during a 60-event stream, then end the drain early with a second SIGTERM.
U=http://127.0.0.1:${1:-18490}
ts() { date +%T.%3N; }
stamp() { while IFS= read -r l; do [ -n "$l" ] && echo "$(ts) $1 $l"; done; }
{ curl -sSN "$U/stream?n=60" -w 'http=%{http_code}\n' 2>&1; echo "curl rc=$?"; } | stamp A &
sleep 2.5
echo "$(ts) restart begins"
(systemctl --user restart tsvdemo-drain.service; echo "$(ts) restart returned rc=$?") &
sleep 3
echo "$(ts) second SIGTERM to the main process"
systemctl --user kill --kill-whom=main --signal=SIGTERM tsvdemo-drain.service
wait

    $ chmod +x abort-test.sh
$ ./abort-test.sh
14:21:31.564 A data: pid=2190293 event=1/60
14:21:32.565 A data: pid=2190293 event=2/60
14:21:33.566 A data: pid=2190293 event=3/60
14:21:34.059 restart begins
14:21:34.566 A data: pid=2190293 event=4/60
14:21:35.566 A data: pid=2190293 event=5/60
14:21:36.567 A data: pid=2190293 event=6/60
14:21:37.062 second SIGTERM to the main process
14:21:37.069 A curl: (18) transfer closed with outstanding read data remaining
14:21:37.070 A http=200
14:21:37.071 A curl rc=18
14:21:37.072 restart returned rc=0

    Expected: A is cut (curl: (18)) 3.0 s into the drain, and the restart returns 10 ms after the second signal. The journal records exit status 1 as Failed with result 'exit-code', then starts the new process on the socket:

    $ journalctl --user -u tsvdemo-drain.service --since -15s -o short-precise
[…]
Oct 25 14:21:34.065330 host systemd[1397]: Stopping tsvdemo-drain.service - tsvdemo-drain: SSE demo that drains running requests on stop...
Oct 25 14:21:34.065848 host tsvdemo-drain[2190293]: 2026/10/25 14:21:34 shutdown: draining for up to 2m0s
Oct 25 14:21:37.067622 host systemd[1397]: tsvdemo-drain.service: Sent signal SIGTERM to main process 2190293 (tsvdemo-drain) on client request.
Oct 25 14:21:37.068169 host tsvdemo-drain[2190293]: 2026/10/25 14:21:37 shutdown: second signal, exiting now
Oct 25 14:21:37.068962 host systemd[1397]: tsvdemo-drain.service: Main process exited, code=exited, status=1/FAILURE
Oct 25 14:21:37.069055 host systemd[1397]: tsvdemo-drain.service: Failed with result 'exit-code'.
Oct 25 14:21:37.069645 host systemd[1397]: Stopped tsvdemo-drain.service - tsvdemo-drain: SSE demo that drains running requests on stop.
Oct 25 14:21:37.071006 host systemd[1397]: Started tsvdemo-drain.service - tsvdemo-drain: SSE demo that drains running requests on stop.
Oct 25 14:21:37.088652 host tsvdemo-drain[2190806]: 2026/10/25 14:21:37 pid 2190806 serving on 127.0.0.1:18490 (inherited=true)

  10. Apply later changes with install-units.sh. After an edit to the service file only (here its Description=), the socket file is unchanged: the script stops the service behind the open socket and then starts both, so a request made meanwhile queues as in step 7:

    $ sed -i 's/^Description=tsvdemo-drain:/Description=tsvdemo-drain (edited):/' tsvdemo-drain.service && ./restart-test.sh 18490 ./install-units.sh
14:21:37.604 A data: pid=2190806 event=1/15
14:21:38.605 A data: pid=2190806 event=2/15
14:21:39.605 A data: pid=2190806 event=3/15
14:21:40.606 A data: pid=2190806 event=4/15
14:21:41.098 restart begins
socket file unchanged: restarting the service behind the open socket
14:21:41.606 A data: pid=2190806 event=5/15
14:21:42.608 A data: pid=2190806 event=6/15
14:21:43.101 B starts
14:21:43.609 A data: pid=2190806 event=7/15
14:21:44.149 ss LISTEN 1      4096   127.0.0.1:18490 0.0.0.0:*
14:21:44.610 A data: pid=2190806 event=8/15
14:21:45.610 A data: pid=2190806 event=9/15
14:21:46.610 A data: pid=2190806 event=10/15
14:21:47.611 A data: pid=2190806 event=11/15
14:21:48.613 A data: pid=2190806 event=12/15
14:21:49.614 A data: pid=2190806 event=13/15
14:21:50.614 A data: pid=2190806 event=14/15
14:21:51.615 A data: pid=2190806 event=15/15
14:21:51.616 A http=200
14:21:51.617 A curl rc=0
Stopping 'tsvdemo-drain.service', but its triggering units are still active:
tsvdemo-drain.socket
14:21:51.727 restart returned rc=0
14:21:51.737 B data: pid=2192112 event=1/1
14:21:51.738 B http=200 connect=0.000108s total=8.627194s
14:21:51.739 B curl rc=0

    Expected: socket file unchanged: …, systemd's warning that the socket is still active (that is the point here), B queued and answered by the new PID.

    After an edit to the socket file (here the address, in both files), the script stops socket and service in one call before it starts both:

    $ sed -i 's/127.0.0.1:18490/127.0.0.1:18491/' tsvdemo-drain.socket tsvdemo-drain.service && ./install-units.sh
socket file changed: stopping socket and service, then starting both
$ ss -ltnp '( sport = :18490 or sport = :18491 )'; journalctl --user -u tsvdemo-drain.service -g serving -n 1 -o cat
State  Recv-Q Send-Q Local Address:Port  Peer Address:PortProcess                                    
LISTEN 0      4096       127.0.0.1:18491      0.0.0.0:*    users:(("tsvdemo-drain",pid=2192375,fd=5))
2026/10/25 14:21:52 pid 2192375 serving on 127.0.0.1:18491 (inherited=true)

    Expected: socket file changed: …, one LISTEN line on 18491, and inherited=true on 18491.

    NoteA socket change refuses or resets new connections until the drain ends

    In a separate run of the same move (18490 to 18491) with a stream running, two requests made during the drain, one to each port, both failed at once with curl: (7) … Could not connect to server, while the stream on the old port finished 8 of 8. With the address unchanged and another line of the socket file edited, a request made during the drain connected, waited, and got curl: (56) Recv failure: Connection reset by peer when the socket closed after the service. A plain restart (steps 7–9) has neither problem.

Verify#

The restart test on the final address passes: A complete, B connected at once and served by the new PID, one connection queued during the drain.

$ ./restart-test.sh 18491
14:21:52.937 A data: pid=2192375 event=1/15
14:21:53.937 A data: pid=2192375 event=2/15
14:21:54.939 A data: pid=2192375 event=3/15
14:21:55.940 A data: pid=2192375 event=4/15
14:21:56.432 restart begins
14:21:56.941 A data: pid=2192375 event=5/15
14:21:57.941 A data: pid=2192375 event=6/15
14:21:58.435 B starts
14:21:58.941 A data: pid=2192375 event=7/15
14:21:59.479 ss LISTEN 1      4096   127.0.0.1:18491 0.0.0.0:*
14:21:59.942 A data: pid=2192375 event=8/15
14:22:00.943 A data: pid=2192375 event=9/15
14:22:01.943 A data: pid=2192375 event=10/15
14:22:02.943 A data: pid=2192375 event=11/15
14:22:03.945 A data: pid=2192375 event=12/15
14:22:04.945 A data: pid=2192375 event=13/15
14:22:05.947 A data: pid=2192375 event=14/15
14:22:06.947 A data: pid=2192375 event=15/15
14:22:06.948 A http=200
14:22:06.949 A curl rc=0
14:22:07.041 restart returned rc=0
14:22:07.060 B data: pid=2193535 event=1/1
14:22:07.061 B http=200 connect=0.000109s total=8.615904s
14:22:07.062 B curl rc=0

Undo / rollback#

  • Drop socket activation and keep the service (the path back for your own service): stop and remove the socket unit, remove the Wants=/After= lines, reload, restart. The server binds the address itself again:
    $ systemctl --user disable --runtime --now tsvdemo-drain.socket && rm "$UNITDIR/tsvdemo-drain.socket" && sed -i "/tsvdemo-drain.socket/d" "$UNITDIR/tsvdemo-drain.service" && systemctl --user daemon-reload && systemctl --user restart tsvdemo-drain.service && sleep 1 && journalctl --user -u tsvdemo-drain.service -g serving -n 1 -o cat
Removed '/run/user/1000/systemd/user/sockets.target.wants/tsvdemo-drain.socket'.
2026/10/25 14:22:09 pid 2193892 serving on 127.0.0.1:18491 (inherited=false)
    What that gives back: a request during a restart is refused (curl: (7)), and ss prints nothing during the drain because nothing listens:
    $ ./restart-test.sh 18491
14:22:10.190 A data: pid=2193892 event=1/15
14:22:11.190 A data: pid=2193892 event=2/15
14:22:12.191 A data: pid=2193892 event=3/15
14:22:13.193 A data: pid=2193892 event=4/15
14:22:13.684 restart begins
14:22:14.194 A data: pid=2193892 event=5/15
14:22:15.194 A data: pid=2193892 event=6/15
14:22:15.688 B starts
14:22:15.697 B curl: (7) Failed to connect to 127.0.0.1 port 18491 after 0 ms: Could not connect to server
14:22:15.699 B http=000 connect=0.000000s total=0.000153s
14:22:15.700 B curl rc=7
14:22:16.195 A data: pid=2193892 event=7/15
14:22:17.196 A data: pid=2193892 event=8/15
14:22:18.196 A data: pid=2193892 event=9/15
14:22:19.197 A data: pid=2193892 event=10/15
14:22:20.198 A data: pid=2193892 event=11/15
14:22:21.199 A data: pid=2193892 event=12/15
14:22:22.201 A data: pid=2193892 event=13/15
14:22:23.201 A data: pid=2193892 event=14/15
14:22:24.202 A data: pid=2193892 event=15/15
14:22:24.203 A http=200
14:22:24.204 A curl rc=0
14:22:24.260 restart returned rc=0
  • Return your own service to its earlier unit file: copy ~/<APP>.service.orig back to ~/.config/systemd/user/<APP>.service, remove its .socket unit as in the first bullet, run systemctl --user daemon-reload and restart the service. [UNVERIFIED: not exercised; the demo had no earlier unit file]
  • Remove the demo completely, then delete the demo directory:
    $ systemctl --user disable --runtime --now tsvdemo-drain.service && rm "$UNITDIR/tsvdemo-drain.service" && systemctl --user daemon-reload && systemctl --user list-units --all "tsvdemo-*"
Removed '/run/user/1000/systemd/user/default.target.wants/tsvdemo-drain.service'.
  UNIT LOAD ACTIVE SUB DESCRIPTION

0 loaded units listed.
To show all installed unit files use 'systemctl list-unit-files'.
  • For a permanent install, the same commands without --runtime, with UNITDIR = ~/.config/systemd/user. [UNVERIFIED: rollback not exercised outside the runtime unit directory]

Failure modes#

Symptom (exact message)CauseFix
curl: (18) transfer closed with outstanding read data remaining as soon as a stop begins; the log shows shutdown: draining for up to 2m0s and exit in the same secondmain returns when Serve returns ErrServerClosed, while Shutdown still drains in its goroutineWait for Shutdown before returning (serveUntil, step 1)
child failed: signal: terminated, HTTP 502, as soon as a restart beginsKillMode=control-group (the default) sends SIGTERM to every process in the unitKillMode=mixed (step 4)
State 'stop-sigterm' timed out. Aborting., Killing process 1972107 (tsvdemo-drain) with signal SIGABRT., a Go SIGABRT: abort goroutine dump, Failed with result 'timeout'TimeoutStopSec= below the drain (a 5 s drop-in in my run). Fedora 38 and later ship /usr/lib/systemd/user/service.d/10-timeout-abort.conf with TimeoutStopFailureMode=abort, so the timeout sends SIGABRT before any SIGKILL; systemd.service(5) gives terminate as the defaultTimeoutStopSec= above -drain (step 4)
shutdown: drain time over (context deadline exceeded), closing and stream: cut after 9/15 events, then curl: (18)The request ran longer than -drain (5 s in my run)Working as designed; raise -drain and TimeoutStopSec= together if such requests must finish
Stopping 'tsvdemo-drain.service', but its triggering units are still active: tsvdemo-drain.socket; the next request makes the service active againThe socket still listens and starts the service on the next connectionStop both: systemctl --user stop tsvdemo-drain.socket tsvdemo-drain.service
tsvdemo-drain.socket: Unit configuration changed while unit was running, and no socket file descriptors are open. Unit not functional until restarted.; after a service restart the log says inherited=falseListenStream= changed and daemon-reload ran while the socket was active (the file was copied by hand, so install-units.sh then sees no change); is-active still says activesystemctl --user stop tsvdemo-drain.socket tsvdemo-drain.service, then systemctl --user start tsvdemo-drain.socket tsvdemo-drain.service
Job failed. See "journalctl -xe" for details. from systemctl --user restart tsvdemo-drain.socket; journal: Socket service tsvdemo-drain.service already active, refusing.The socket unit cannot start while its service runsSame as the row above
systemd passed "2" sockets, want 1 (one ListenStream= line), the service exits 1 and is-active says activating (waiting for Restart=)Two ListenStream= lines (in my run the second came from a drop-in)One ListenStream= in the socket unit and its drop-ins
Failed to parse address value in ':18490', ignoring: Invalid argument and Unit has no Listen setting (ListenStream=, ListenDatagram=, ListenFIFO=, ...). Refusing. (the same for localhost:18490)systemd reads an IP literal with a port, a bare port and other forms listed in systemd.socket(5), not Go's :port or a host nameListenStream=127.0.0.1:18490; install-units.sh runs systemd-analyze --user verify before it stops anything
With Requires=: A dependency job for tsvdemo-drain.service failed. See 'journalctl -xe' for details.; journal: Failed to create listening socket (127.0.0.1:18490): Address already in use, Dependency failed for tsvdemo-drain.service; still down after the port is freeRequires= fails the service when the socket fails; no service process ran, so Restart= never firesWants= (step 4): the server's own bind fails (listen tcp 127.0.0.1:18490: bind: address already in use), Restart= retries every 5 s, and the first retry after the port is free gets the socket
curl: (7) Failed to connect to 127.0.0.1 port 18491 after 0 ms: Could not connect to server during a restartNo socket unit: the old process closes its listener when the drain starts, and the new one binds after the drainThe socket unit (steps 3–6)

Field notes#

  • 2026-10-25 13:38–14:22, GEEKOM AE8, Fedora Linux 44, kernel 7.2.10, systemd 259 (259.9-1.fc44), Go 1.26.0, curl 8.18.0, iproute2 6.17.0: the demo above. Units in $XDG_RUNTIME_DIR/systemd/user, enabled with --runtime; the demo directory sat in my writing workspace, its files saved with mode 644; ports 127.0.0.1:18490–18493. Every block above comes from one run in step order (14:20:56–14:22:25), and so do the 502 and curl: (7) rows of Failure modes; the other rows and the note in step 10 come from two more runs (13:41–13:42 and 13:46–13:47).
    • I ran the step sequence four times. In the first pass curl -s hid curl's error texts, and the Undo test started its stream right after systemctl restart had returned: the self-binding server was not listening yet and the stream got curl rc=7. A server that binds by itself refuses connections right after its start too, not only during a drain. The second pass still had a unit comment that promised SIGKILL at the stop timeout (wrong on Fedora, see below), and the unchanged-socket branch of install-units.sh had never run; the third pass fixed both. The first three copied the files with their executable bits, so a chmod that named the test scripts before they existed went unnoticed; the fourth saved everything with mode 644, as from this page, and ran the chmod lines of steps 5 and 7–9.
    • My first port holder for the Wants=/Requires= cases, a Python socket without SO_REUSEADDR, failed to bind 18490 at all (Address already in use), most likely because connections from earlier tests were in TIME_WAIT on that port (I did not capture them). With SO_REUSEADDR set it held the port and the socket unit failed as intended.
    • The stop timeout case ended with SIGABRT, not the SIGKILL systemd.service(5) describes for the default TimeoutStopFailureMode=terminate. The abort comes from Fedora's drop-in /usr/lib/systemd/user/service.d/10-timeout-abort.conf (package systemd-259.9-1.fc44), part of the Fedora 38 "Shorter Shutdown Timer" change, which also set the 45 s default stop timeout. The Go runtime answered SIGABRT with a 221-line goroutine dump in the journal and exit status 2.
    • ss -ltnp never listed the systemd user manager as a holder of the socket, only the server process: as my user, reading the links in /proc/<manager PID>/fd/ fails with Permission denied (154 of 154), so ss cannot attribute the manager's copy. During a drain the port showed as LISTEN with a queued connection and no process.
    • After a ListenStream= change copied in by hand plus daemon-reload, systemctl is-active still said active for a socket without file descriptors. Only the server's inherited=false log line showed it.
    • My first install-units.sh handled an unchanged socket file with start of the socket and then restart of the service. In the state from the previous point, start of the socket failed with "already active, refusing" and set -e ended the script. The published version stops the service first, as the Makefile of the service this comes from does.
    • go mod init with Go 1.26.0 wrote go 1.25.0 into go.mod.
    • At 14:28 the same day a system upgrade installed systemd 259.10-1.fc44, after all of the runs above; nothing here was re-run on 259.10.
    • The job lines in step 2's Expected come from a separate paste of that line into an interactive bash on a pseudo-terminal driven by a script (14:12); everything else in step 2 is from the step-order run.
  • 2026-10-25 00:45–02:03, same box, systemd 259 (259.9-1.fc44, installed 2026-09-21), a Go LLM routing proxy (the service this manual comes from; the Go version is not in my notes of these runs), throwaway units on 127.0.0.1:18471: five rounds of tests of the change that introduced the socket unit and the drain. The default KillMode sent SIGTERM to the ssh call of an engine switch when a stop began, and the client got an error after 4 s; with KillMode=mixed the switch finished and the client got its full stream (62 s). A ListenStream= change with only a daemon-reload left the socket without file descriptors, and the proxy silently bound the port itself. A stopped service came back on the next client poll through the socket. Requires= turned a socket bind failure into a service that stayed down; Wants= let Restart= recover. The previous release's 5 s shutdown grace had never run, because main returned as soon as ListenAndServe returned ErrServerClosed.

Notes#

  • The story of how the proxy got here: Shutdown grace that never ran. For llama-server units, KillMode and a SIGTERM to the main process only: Terminating immediately took 90 seconds.
  • Requests that arrive during a restart wait for the whole drain (B waited 9.0 s in step 7). A client whose timeout is shorter than the drain gives up. How many connections can wait is the socket's Backlog=; systemd.socket(5) gives its default as 4294967295, capped by net.core.somaxconn (4096 here).
  • net/http's Shutdown "does not attempt to close nor wait for hijacked connections such as WebSockets", so serveUntil does not drain them. Not tested here.
  • systemd 259 also sets LISTEN_PIDFDID, and sd_listen_fds(3) checks it against the process's pidfd; listenerFrom checks only LISTEN_PID.
  • fd 3 arrives without close-on-exec: a probe under systemd-socket-activate, which passes sockets the same way, read FD_CLOEXEC unset. net.FileListener works on a close-on-exec duplicate (poll.DupCloseOnExec in net/file_unix.go), so closing fd 3 keeps the socket out of children (step 8: fds 0, 1, 2 only).
  • github.com/coreos/go-systemd/v22/activation (v22.7.0) does the descriptor part; at v22.7.0 its Files(true) unsets LISTEN_PID, LISTEN_FDS and LISTEN_FDNAMES but not LISTEN_PIDFDID. I read its source and did not use it.
  • For a socket on a specific LAN address, systemd.socket(5) recommends FreeBind=true ("for robustness reasons"), so the socket binds before the address is configured. Not tested here.

Filed under: go, systemd, linux, sse

Sources#