cd ..

$ cat ~/field-notes/linux-signals-explained.md

Linux Signals: What SIGTERM, SIGKILL, and SIGHUP Actually Do

kill is badly named. The command does not necessarily kill anything—it sends a signal. What happens next depends on the signal and the receiving process.

Understanding that distinction explains graceful shutdown, terminal shortcuts, daemon reloads, zombie processes, and a surprising number of broken Docker entrypoints.

A Signal Is a Notification

A signal interrupts a process to report an event. Each signal has a number, a name, and a default action.

kill -l               # list signals on this system
kill -TERM 1234       # send SIGTERM to PID 1234
kill -HUP 1234        # send SIGHUP
kill -0 1234          # send nothing; check existence and permission

Applications can install handlers for most signals. They may clean up, reload configuration, log a diagnostic, or ignore the signal.

Two signals are special: SIGKILL and SIGSTOP cannot be caught, blocked, or ignored. The kernel enforces them directly.

The Signals You Actually Use

SIGTERM  polite shutdown request; default action is termination
SIGINT   interactive interrupt, usually Ctrl-C
SIGKILL  immediate forced termination; cannot be handled
SIGHUP   terminal disconnected; commonly repurposed as config reload
SIGQUIT  terminal quit, often produces a core dump
SIGUSR1  application-defined signal
SIGUSR2  application-defined signal
SIGCHLD  a child process stopped or exited
SIGPIPE  wrote to a pipe/socket with no reader
SIGSTOP  pause immediately; cannot be handled
SIGCONT  resume a stopped process

Signal meanings are conventions layered on kernel defaults. nginx uses SIGHUP to reload configuration. Another program may ignore it or terminate.

Read the application's documentation before sending signals in automation.

SIGTERM vs SIGKILL

SIGTERM starts a conversation:

orchestrator → SIGTERM → application
application:
  stop accepting work
  finish in-flight requests
  flush buffered data
  close connections
  exit

SIGKILL ends the conversation:

kernel stops scheduling the process immediately
no finally blocks
no deferred cleanup
no buffer flush
no application log saying why it stopped

This is why kill -9 should be the last step, not the first. Files are not automatically corrupted by SIGKILL, but any application-level invariants or buffered writes in progress lose their chance to finish.

A sensible shutdown sequence is:

kill -TERM "$pid"

# Wait up to 15 seconds.
for _ in $(seq 1 150); do
    kill -0 "$pid" 2>/dev/null || exit 0
    sleep 0.1
done

kill -KILL "$pid"

Supervisors and container orchestrators implement the same pattern with a configurable grace period.

Handling Signals in a Shell Script

#!/usr/bin/env bash
set -euo pipefail

child_pid=''

shutdown() {
    echo 'forwarding shutdown to child'
    if [[ -n "$child_pid" ]]; then
        kill -TERM "$child_pid" 2>/dev/null || true
        wait "$child_pid"
    fi
}

trap shutdown TERM INT

./server &
child_pid=$!
wait "$child_pid"

Without the trap, the shell can receive SIGTERM while the actual server keeps running. Signal forwarding is one reason wrapper scripts deserve more care than a one-line server & wait.

If the script does not need setup or teardown, replace the shell entirely:

exec ./server

exec makes the server inherit the script's PID, so signals go directly to it.

PID 1 Is Different

Inside a container, the configured command usually becomes PID 1. PID 1 has two extra responsibilities:

  1. Reap orphaned child processes so they do not remain zombies
  2. Handle or forward signals to the process tree

This Dockerfile form starts the program directly:

# Good: exec form, server receives signals
CMD ["node", "server.js"]

This form inserts a shell:

# Risky: /bin/sh is PID 1 and may not forward the signal
CMD node server.js

Use exec-form commands. If the application creates child processes and does not reap them, use a tiny init process such as tini or the runtime's init option.

Process Groups Matter

Sending a signal to one PID does not automatically signal its children.

# Signal one process.
kill -TERM 1234

# Signal the entire process group whose ID is 1234.
kill -TERM -- -1234

Build tools, test runners, and shell pipelines often create process trees. Killing only the parent can leave workers holding ports and temporary files. Supervisors normally manage a process group or cgroup for this reason.

SIGHUP and Configuration Reloads

Traditional daemons were attached to a terminal. When that terminal disappeared, the kernel sent SIGHUP—hangup. Daemons later adopted the signal as a reload convention.

A safe reload looks like this:

receive SIGHUP
read new configuration
validate the entire configuration
if valid: atomically replace current config
if invalid: log error and keep old config

Do not partially apply a broken configuration. And remember that environment variables cannot be changed inside an already running process by editing the parent's environment; a reload must read a file or another external source.

SIGCHLD and Zombie Processes

When a child exits, the kernel keeps a small status record until the parent calls wait(). That finished-but-uncollected entry is a zombie.

ps -eo pid,ppid,state,command
# State Z means zombie.

SIGCHLD tells the parent that child state changed. A server that forks children must reap them. Killing the zombie does nothing—it is already dead. Fix or restart the parent that failed to call wait().

Debugging Signals

# Watch signal-related system calls for an existing process.
strace -f -e trace=signal -p 1234

# Show signal masks and handlers exposed by the kernel.
cat /proc/1234/status | grep '^Sig'

# Ask a process to dump state if the application defines SIGUSR1 for that.
kill -USR1 1234

Never assume SIGUSR1 is harmless. "User-defined" means the application decides what it does.

The Bottom Line

Signals are a tiny control plane for Unix processes.

The rules:

  • Send SIGTERM first and allow a bounded grace period
  • Use SIGKILL only when the process will not cooperate
  • Use exec in simple entrypoint scripts
  • Forward signals when a wrapper owns child processes
  • Use exec-form container commands
  • Reap children when running as PID 1
  • Treat SIGHUP and SIGUSR signals as application-specific APIs
  • Signal process groups when you intend to stop a whole tree

Once you see kill as "send a process event," the commands and failure modes make much more sense.