cd ..

$ cat ~/field-notes/health-checks-done-right.md

Health Checks Done Right: Liveness, Readiness, and Startup

5 min read 0x.run

GET /health returning {"ok": true} feels responsible. It is only useful if you know who calls it and what that caller will do with the answer.

A load balancer may stop traffic. An orchestrator may restart the process. A human may open a diagnostic page. Those are different decisions and need different signals.

One Endpoint Cannot Answer Every Question

Use three concepts:

liveness  — is this process stuck in a way a restart might fix?
readiness — should this instance receive new traffic right now?
startup   — has slow initialization completed yet?

Conflating them creates dangerous feedback loops. If a database outage makes liveness fail, every application instance restarts at once. The database remains down, and now the application fleet is cold too.

Dependency failure usually affects readiness, not liveness.

Liveness: Can the Process Make Progress?

A liveness check should be cheap, local, and difficult to fail because of an external outage.

app.get('/health/live', (req, res) => {
    res.json({ status: 'alive' });
});

In many applications, serving this route at all proves the event loop or request threads are alive. More advanced checks may detect a permanently deadlocked worker, exhausted critical internal queue, or failed background supervisor.

Do not query the database, Redis, DNS, payment provider, and three internal services from liveness. Restarting this process will not repair those systems.

Readiness: Should Traffic Arrive?

Readiness represents the instance's ability to serve its intended workload.

const state = {
    acceptingTraffic: false,
    databaseReady: false,
};

app.get('/health/ready', (req, res) => {
    const ready = state.acceptingTraffic && state.databaseReady;
    res.status(ready ? 200 : 503).json({
        status: ready ? 'ready' : 'not_ready',
    });
});

Set acceptingTraffic only after configuration loads, migrations or compatibility checks complete, and required connection pools can be initialized. Set it false at the beginning of graceful shutdown so load balancers drain the instance before the process exits.

Whether a dependency belongs in readiness depends on the service contract. If every request requires the primary database, loss of database connectivity makes the instance unready. If only an optional recommendation panel uses Redis, Redis failure should degrade that feature rather than remove the entire instance from service.

Startup: Give Initialization Its Own Window

Some applications need time to load models, warm caches, replay a log, or compile templates. A strict liveness check can kill them before startup completes.

Kubernetes supports a startup probe that runs until it succeeds:

startupProbe:
  httpGet:
    path: /health/startup
    port: 8080
  periodSeconds: 2
  failureThreshold: 60  # up to 120 seconds

livenessProbe:
  httpGet:
    path: /health/live
    port: 8080
  periodSeconds: 10
  failureThreshold: 3

readinessProbe:
  httpGet:
    path: /health/ready
    port: 8080
  periodSeconds: 5
  failureThreshold: 2

Until startup succeeds, liveness and readiness do not need to race slow initialization. After it succeeds, normal health policy takes over.

Avoid the Dependency Cascade

Imagine 100 API instances check the database every second. The database slows down. Health queries add load, readiness fails everywhere, and the platform removes all instances from service. Users receive no response even though some requests could have succeeded.

Health checks need the same production discipline as other traffic:

  • Use a tight timeout
  • Keep queries trivial and indexed
  • Reuse connection pools
  • Add a little jitter where the platform allows it
  • Avoid checking third parties from every instance
  • Do not perform writes
  • Do not warm caches or repair state inside the probe

Often the best readiness signal is local state maintained by normal application operations, not a new fan-out on every probe.

Return Little, Observe More

Public health responses should not reveal versions, hostnames, database addresses, exception messages, or environment names.

HTTP/1.1 503 Service Unavailable
Content-Type: application/json

{"status":"not_ready"}

Record detailed reasons in metrics and structured logs:

readiness_state{reason="database_unavailable"} 0
readiness_transition_total{from="ready",to="not_ready"} 1
health_probe_duration_seconds

If operators need a detailed diagnostic endpoint, put it behind authentication and network controls. A health probe is not an admin dashboard.

Use Hysteresis to Stop Flapping

One failed check should not always remove an instance, and one successful check should not always restore it. Failure and success thresholds prevent rapid oscillation.

mark unready after: 2 consecutive failures
mark ready after:   2 consecutive successes

Exact values depend on probe interval and outage tolerance. Too sensitive causes flapping. Too slow routes traffic to an instance that cannot serve it.

The application can also retain dependency state briefly instead of turning every transient packet loss into a fleet-wide readiness transition.

Health During Shutdown

A clean sequence is:

1. receive SIGTERM
2. readiness becomes 503
3. wait for load balancer propagation/drain
4. stop accepting new work
5. finish in-flight requests within deadline
6. close pools and exit

Make sure the platform gives enough grace time for its readiness observation and load balancer update. If the process exits the millisecond it becomes unready, in-flight connections still get cut.

Test the Decision, Not Just the Route

Health checks often work in a browser and fail in reality because the platform uses another port, path, host header, protocol, or timeout.

Test these scenarios in staging:

process deadlock or event-loop stall
database unavailable
optional dependency unavailable
slow startup
graceful shutdown
health endpoint itself times out
recovery after a transient failure

Observe whether the orchestrator restarts, removes, restores, and drains instances as intended. An endpoint unit test cannot validate that control loop.

The Bottom Line

A health check is an input to automation. Design it around the action that automation takes.

The rules:

  • Keep liveness local and fail it only when restart can plausibly help
  • Use readiness to control new traffic
  • Protect slow initialization with a startup check
  • Include only dependencies essential to the instance's contract
  • Keep probes cheap, bounded, and free of side effects
  • Expose minimal public detail; send reasons to logs and metrics
  • Use thresholds to prevent flapping
  • Become unready before graceful shutdown
  • Test the complete orchestrator behavior under failure

The goal is not to produce a green dashboard. It is to help the platform make the safest possible decision during an incident.