"Terminating immediately" took 90 seconds

· 11 min read
Contents

llama-server keeps a message for the moment you run out of patience with it: Received second interrupt, terminating immediately. On the 3090 box it printed that at 09:45:04. The next thing systemd had to say about the unit came at 09:46:34, and it was Aborting.

That stop was half of an engine switch. My LAN proxy for local models stops one llama.cpp build and starts another when a request names a model that only the other build has (why it does that). The proxy logged the new engine up after 1m36s. A switch between those two builds takes 2 to 4 seconds. Two signals did this, the second one came from my own script, and the "immediately" path deadlocked inside glibc.

The 90 seconds in the journal#

The unit's own journal on the 3090 box, hostname replaced:

2026-10-24T09:45:04+03:00 host serve-fork-llama[2941262]: [48275] 0.16.373.173 I srv    operator(): operator(): cleaning up before exit...
2026-10-24T09:45:04+03:00 host systemd[1601]: Stopping llm-engine-fork.service - LLM engine fork on 3090 for llm-router-proxy...
2026-10-24T09:45:04+03:00 host serve-fork-llama[2941262]: Received second interrupt, terminating immediately.
2026-10-24T09:46:34+03:00 host systemd[1601]: llm-engine-fork.service: State 'stop-sigterm' timed out. Aborting.
2026-10-24T09:46:34+03:00 host systemd[1601]: llm-engine-fork.service: Killing process 2941212 (bash) with signal SIGABRT.
2026-10-24T09:46:34+03:00 host systemd[1601]: llm-engine-fork.service: Killing process 2941262 (llama-server) with signal SIGABRT.
2026-10-24T09:46:34+03:00 host systemd[1601]: llm-engine-fork.service: Killing process 2941283 (llama-server) with signal SIGABRT.
[…]
2026-10-24T09:46:34+03:00 host systemd[1601]: llm-engine-fork.service: Main process exited, code=dumped, status=6/ABRT
[…]
2026-10-24T09:46:38+03:00 host systemd[1601]: llm-engine-fork.service: Failed with result 'timeout'.
2026-10-24T09:46:38+03:00 host systemd[1601]: Stopped llm-engine-fork.service - LLM engine fork on 3090 for llm-router-proxy.

Three processes were in the unit's cgroup: bash (2941212, the serve script and the unit's main process), the llama-server router (2941262, which logs under the script's tag) and one child instance, the loaded model on port 48275 (2941283). The child's cleaning up before exit... arrived in the same second as systemd's Stopping: the first SIGTERM had reached the child as well. Then the router announced the second one, and nothing happened for exactly TimeoutStopSec=90.

Two SIGTERMs, and the second one was mine#

systemd's default for stopping a service is KillMode=control-group: "all remaining processes in the control group of this unit will be killed on unit stop". systemctl stop sent SIGTERM to bash, the router and the child at once. That part works as documented.

The serve script added the second one. These are its last seven lines:

"$LLAMA_BIN/llama-server" --host "$HOST" --port "$PORT" \
  --models-preset "$TMP/models.ini" --models-max 1 &
SERVER=$!
# Stopping this script by PID must stop the router too (Ctrl+C reaches both through the process group).
trap 'kill "$SERVER" 2>/dev/null; wait "$SERVER" 2>/dev/null; rm -rf "$TMP"' EXIT
trap 'exit 143' TERM INT HUP
wait "$SERVER"

The comment says what the EXIT trap is for: if someone kills the script by PID, the router must not outlive it. Under systemd that same trap runs alongside systemd's own SIGTERM to the router. bash gets SIGTERM, runs exit 143, the EXIT trap fires, and kill "$SERVER" delivers SIGTERM number two within the same second as number one. Each half is reasonable on its own. Nobody had them both in mind at once, me included.

What "terminating immediately" runs#

Here is the handler in llama.cpp's tools/server/server.cpp, at the commit the engine I call fork was built from. Despite the name it is upstream llama.cpp, a37cc46, newer than the mainline build and needed for models mainline couldn't load yet:

static inline void signal_handler(int signal) {
    if (is_terminating.test_and_set()) {
        // in case it hangs, we can force terminate the server by hitting Ctrl+C twice
        // this is for better developer experience, we can remove when the server is stable enough
        fprintf(stderr, "Received second interrupt, terminating immediately.\n");
        exit(1);
    }

    shutdown_handler(signal);
}

The comment explains the intent, and it is a good one: a server that hangs on shutdown can be killed by pressing Ctrl+C twice. The first signal starts a graceful shutdown. The second calls exit(1), from inside the signal handler.

exit() is not on the signal-safety(7) list. _exit() and _Exit() are. The difference is everything exit() does on the way out: atexit handlers, static destructors, all of it running on top of whatever the interrupted thread was in the middle of. On five of the six engine stops that day, that was nothing that mattered.

The core dump Fedora kept#

The Aborting in the journal, instead of a SIGKILL, comes from a drop-in that Fedora's systemd package ships for user services, /usr/lib/systemd/user/service.d/10-timeout-abort.conf:

# This file is part of the systemd package.
# See https://fedoraproject.org/wiki/Changes/Shorter_Shutdown_Timer.
#
# To facilitate debugging when a service fails to stop cleanly,
# TimeoutStopFailureMode=abort is set to "crash" services that fail to stop in
# the time allotted. This will cause the service to be terminated with SIGABRT
# and a coredump to be generated.
[…]
[Service]
TimeoutStopFailureMode=abort

That made the hang debuggable for free. systemd-coredump stored the router's main thread, as coredumpctl info 2941262 shows it (the first SIGTERM's shutdown at the bottom, the second's on top):

                Stack trace of thread 2941262:
                #0  0x00007feefdb77a69 __lll_lock_wait_private (libc.so.6 + 0x6fa69)
                #1  0x00007feefdb77de0 __GI___nptl_deallocate_stack (libc.so.6 + 0x6fde0)
                #2  0x00007feefdb7cb0e __pthread_clockjoin_ex (libc.so.6 + 0x74b0e)
                #3  0x00007feefdd7f4eb _ZNSt6thread4joinEv (libstdc++.so.6 + 0x514eb)
                #4  0x00007feefe1b80e3 _ZN22stream_session_manager7stop_gcEv (libllama-server-impl.so + 0x1b80e3)
                #5  0x00007feefe1b8164 _ZN22stream_session_managerD2Ev (libllama-server-impl.so + 0x1b8164)
                #6  0x00007feefdb24140 __cxa_finalize (libc.so.6 + 0x1c140)
                #7  0x00007feefe0d3a87 __do_global_dtors_aux (libllama-server-impl.so + 0xd3a87)
                #8  0x00007feefe6662d2 _dl_call_fini (ld-linux-x86-64.so.2 + 0x12d2)
                #9  0x00007feefe66a2ae _dl_fini (ld-linux-x86-64.so.2 + 0x52ae)
                #10 0x00007feefdb246b1 __run_exit_handlers (libc.so.6 + 0x1c6b1)
                #11 0x00007feefdb2478e exit (libc.so.6 + 0x1c78e)
                #12 0x00007feefe0d3ed6 _ZL14signal_handleri (libllama-server-impl.so + 0xd3ed6)
                #13 0x00007feefdb21fb0 __restore_rt (libc.so.6 + 0x19fb0)
                #14 0x00007feefe677bcf __GI__dl_deallocate_tls (ld-linux-x86-64.so.2 + 0x12bcf)
                #15 0x00007feefdb77c72 __nptl_free_stacks (libc.so.6 + 0x6fc72)
                #16 0x00007feefdb77e5d __GI___nptl_deallocate_stack (libc.so.6 + 0x6fe5d)
                #17 0x00007feefdb7cb0e __pthread_clockjoin_ex (libc.so.6 + 0x74b0e)
                #18 0x00007feefdd7f4eb _ZNSt6thread4joinEv (libstdc++.so.6 + 0x514eb)
                #19 0x00007feefe29375b _ZN7httplib10ThreadPool8shutdownEv (libllama-server-impl.so + 0x29375b)
                #20 0x00007feefe0edb3e _ZN19server_http_context4joinEv (libllama-server-impl.so + 0xedb3e)
                #21 0x00007feefe0dbccb _Z12llama_serverR13common_paramsiPPcR12server_child (libllama-server-impl.so + 0xdbccb)
                #22 0x00007feefe0dd231 _Z12llama_serveriPPc (libllama-server-impl.so + 0xdd231)
                #23 0x00007feefdb0b681 __libc_start_call_main (libc.so.6 + 0x3681)
                #24 0x00007feefdb0b798 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x3798)
                #25 0x0000563d31e2d2a5 _start (llama-server + 0x12a5)

Read it from #21 upwards. The main thread was doing the graceful shutdown that the first SIGTERM had asked for: httplib::ThreadPool::shutdown joins a worker thread, and glibc puts that thread's stack back into its stack cache and trims the cache (__nptl_deallocate_stack, __nptl_free_stacks, _dl_deallocate_tls). Frame #13, __restore_rt, is the signal frame: the second SIGTERM landed right there.

The handler called exit(1). exit ran the library destructors (_dl_fini), one of which is the destructor of a process-wide static stream_session_manager g_stream_sessions. Its stop_gc() joins the manager's garbage-collector thread, so glibc goes through the same stack-cache code again, frames #3 to #1, and #0 waits for a lock.

glibc's nptl-stack.c shows which lock. __nptl_deallocate_stack starts with lll_lock (GL (dl_stack_cache_lock), LLL_PRIVATE); and releases it at the end. Frame #16 is that function, holding the lock, interrupted. Frame #1 is the same function on the same thread, asking for it again. lll_lock is not recursive. The thread waited for itself until systemd ran out of patience too. bash's own core dump from the same SIGABRT has it inside run_exit_trap, still in wait "$SERVER". (The source I read is glibc 2.42; the box runs 2.43-9.fc44, and the frames follow the same path.)

Five of six stops got lucky#

Every engine-unit stop on the 3090 box that day before the fix, from its journal:

TimeEngineSecond interruptUnit stopped after
01:53:12mainlineyessame second, status=1/FAILURE
01:53:31forkyes1 s, status=1/FAILURE
01:54:24mainlineyessame second, status=1/FAILURE
09:44:44mainlineyessame second, status=1/FAILURE
09:45:04forkyes94 s, status=6/ABRT, three core dumps
09:47:59mainlineyessame second, status=1/FAILURE

All six got the double signal. The fork build survived it at 01:53:31, and the mainline build (f95b0d9, by its build directory's BUILD_REF) has the same handler and the same g_stream_sessions destructor; I checked its server.cpp and server-stream.cpp. Neither build is the guilty one. What differs is where the main thread happened to be standing when the second signal arrived. Inside glibc's stack cache, it never left. Anywhere else, exit(1) worked, and four clean stops before the hang had looked like proof that stopping worked.

Signal the script, not the cgroup#

The fix went into the proxy's host-side helper (the same binary, run over ssh on the GPU box), which is what stops engine units. It signals only the unit's main process, gives it 15 seconds, and only then falls back to a full stop:

// GracefulStop is how long stopUnit waits after signalling only the main process.
const GracefulStop = 15 * time.Second

// stopUnit signals only the unit's main process (the serve script, whose trap stops its server once),
// then falls back to systemctl stop. A plain stop sends SIGTERM to the whole cgroup at once, so the
// script's trap hits llama-server a second time, and the fork build can hang on that until
// TimeoutStopSec (2026-10-24: 90 s, then SIGABRT).
func (a *Agent) stopUnit(u string) error {
	if _, err := a.systemctl("kill", "--kill-whom=main", "--signal=SIGTERM", u); err == nil {
		for waited := time.Duration(0); waited < GracefulStop; waited += 250 * time.Millisecond {
			s, _ := a.systemctl("is-active", u)
			if s = strings.TrimSpace(s); s != "active" && s != "activating" && s != "deactivating" {
				return nil
			}
			a.Sleep(250 * time.Millisecond)
		}
	}
	if _, err := a.systemctl("stop", u); err != nil {
		return fmt.Errorf("systemctl stop %s: %w", u, err)
	}
	return nil
}

bash gets the only SIGTERM. Its trap sends llama-server exactly one. The first stop with it, at 09:48:39:

2026-10-24T09:48:39+03:00 host systemd[1601]: llm-engine-fork.service: Sent signal SIGTERM to main process 2943035 (bash) on client request.
2026-10-24T09:48:39+03:00 host serve-fork-llama[2943083]: 0.17.578.524 I srv    operator(): operator(): cleaning up before exit...
2026-10-24T09:48:39+03:00 host serve-fork-llama[2943083]: 0.17.578.671 I srv    unload_all: stopping model instance name=flint-40b
2026-10-24T09:48:39+03:00 host serve-fork-llama[2943083]: [45835] 0.15.925.600 I srv    operator(): exit command received, exiting...
2026-10-24T09:48:39+03:00 host serve-fork-llama[2943083]: [45835] 0.15.925.657 I srv    operator(): operator(): cleaning up before exit...
2026-10-24T09:48:40+03:00 host serve-fork-llama[2943083]: 0.18.484.017 I srv           run: instance name=flint-40b exited with status 0
2026-10-24T09:48:40+03:00 host systemd[1601]: llm-engine-fork.service: Main process exited, code=exited, status=143/n/a
2026-10-24T09:48:40+03:00 host systemd[1601]: llm-engine-fork.service: Failed with result 'exit-code'.

One second from signal to exit, and the child instance shut down cleanly (exited with status 0). The comment in the fix blames the fork build; the journal and the shared source say either build could have hung. The next four stops, up to 09:50:07, were the same, with no second interrupt in the journal, and the proxy's switches went back to 2-4 seconds. systemd calls the result Failed with result 'exit-code' because the trap exits with 143. Cosmetic, and true.

Two other ways out that I didn't run: KillMode=mixed, which per systemd.kill(5) sends the SIGTERM to the main process only and the SIGKILL that follows to whatever is left in the cgroup, or a wrapper that execs the server so there is one process to signal. The units later dropped the wrapper anyway and run llama-server directly. The script's own comment also says Ctrl+C reaches both processes through the process group, so a Ctrl+C in a terminal sets up the same two signals. I haven't timed that one.

The comment above that exit(1) says it can go once the server is stable enough. In all five stops after the fix, with one signal each, it shut down cleanly.

Filed under: linux, systemd, llama.cpp, bash, local-llm

Sources#