Zero-Downtime Deployments: Rolling, Blue-Green, and Canary

Zero-downtime deployment does not mean no process restarts. It means users continue receiving valid responses while instances are replaced.

The traffic switch is the visible part. Version compatibility, readiness, draining, and rollback decide whether it actually works.

The Non-Negotiable Prerequisites

Every safe strategy needs these properties:

multiple instances behind a traffic manager
readiness signal that reflects ability to serve
graceful shutdown and connection draining
old and new versions able to run simultaneously
backward-compatible database and message changes
enough spare capacity during replacement
observable health and a rollback procedure

If version 2 starts by dropping a column version 1 still reads, no load balancer algorithm can save the deployment.

Rolling Deployment

A rolling update replaces instances gradually.

start:     [v1] [v1] [v1] [v1]
step 1:    [v2] [v1] [v1] [v1]
step 2:    [v2] [v2] [v1] [v1]
step 3:    [v2] [v2] [v2] [v1]
complete:  [v2] [v2] [v2] [v2]

Advantages:

Uses existing fleet capacity efficiently
Built into most orchestrators
Stops automatically if new instances never become ready
Works well for frequent, low-risk changes

Costs:

Old and new versions coexist for the whole rollout
Rollback is another rolling deployment
A bad version can reach meaningful traffic before alarms fire
Reducing capacity during replacement can overload remaining instances

Control surge and unavailability explicitly. For a four-instance service, allowing one extra instance and zero unavailable instances needs capacity for five during the rollout but preserves serving capacity.

Blue-Green Deployment

Blue-green keeps two complete environments. Blue serves production while green receives the new version.

before switch:
traffic → blue [v1 v1 v1 v1]
          green [v2 v2 v2 v2]  (validation)

after switch:
          blue [v1 v1 v1 v1]  (standby)
traffic → green [v2 v2 v2 v2]

Advantages:

Traffic switches as one controlled operation
Old environment remains available for fast rollback
Green can be smoke-tested before receiving users
Clear separation between versions

Costs:

Requires nearly double compute capacity during deployment
Environment drift can make green validation misleading
Stateful services and long-lived connections complicate the switch
Database changes are still shared unless the data layer is duplicated and synchronized

Do not destroy blue immediately. Keep it until green has passed the observation window and rollback is no longer needed. But do not keep it indefinitely either; an old environment becomes unpatched, stale, and expensive.

Canary Deployment

A canary sends a small percentage of real traffic to the new version.

95% → v1 fleet
 5% → v2 canary

then: 75/25 → 50/50 → 0/100

Advantages:

Limits the initial blast radius
Tests production traffic, dependencies, and data shapes
Supports automated promotion based on observed behavior
Reveals problems synthetic smoke tests miss

Costs:

Requires traffic splitting and good version-level metrics
Low traffic can take a long time to produce confidence
A 1% canary can still affect every user if it corrupts shared data
Comparing unequal user populations can produce misleading results

A canary limits request exposure, not necessarily side-effect exposure. A new worker that writes malformed events to a shared topic can damage the whole system while handling only a few requests.

Choose Canary Signals Before Deploying

Do not stare at a dashboard and improvise whether the canary "looks fine." Define promotion and rollback criteria first.

rollback if, compared with stable:
  error rate increases by more than threshold
  p95/p99 latency exceeds threshold
  success or conversion rate decreases materially
  resource use per request exceeds safe capacity
  dependency failures or queue age increase
  critical correctness check fails once

Compare canary to stable at the same time. A global traffic spike can make both slower; a regional dependency problem can affect only one pool.

Use enough requests and time to cover important paths, but cap the observation window so deployments do not remain half-finished forever.

Readiness Before Traffic

An instance starting successfully does not mean it is ready.

Startup may require:

configuration validation
connection pool initialization
schema compatibility check
route registration
cache or model loading
background consumer coordination

Only add the instance to load balancing after readiness succeeds. If startup fails, keep the old capacity serving.

Readiness must also become false during shutdown. Traffic systems need time to observe that state before the process exits.

Drain Long-Lived Work

For normal HTTP requests:

1. mark instance unready
2. stop accepting new connections
3. allow in-flight requests to finish
4. stop background consumers from claiming work
5. checkpoint or return unfinished jobs safely
6. close pools and exit before termination deadline

WebSockets, server-sent events, uploads, and long polling can outlive a typical deployment grace period. Options include longer drain windows, client reconnect instructions, or a protocol that lets the client resume on another instance.

Do not wait forever. Every drain needs a deadline and a safe forced-termination path.

Data Compatibility Is the Hard Part

During a rollout, both versions read and write the same database and messages. Use expand-and-contract changes:

1. expand: add new nullable column or new message field
2. deploy code that can read old and new shapes
3. write/backfill the new shape
4. verify old code is gone and data is complete
5. contract: remove old column or field in a later deployment

Readers must ignore unknown message fields. Producers should not make a new field mandatory until every consumer understands it.

Rollback means running old code against data written by new code. Test that direction too. A deployment is not reversible merely because the old image still exists.

Feature Flags Separate Deploy from Release

A feature flag can deploy dormant code to the full fleet, then enable behavior for internal users or a percentage of traffic.

This is useful when traffic routing cannot isolate the feature itself. It also provides a fast behavioral off switch without replacing binaries.

Flags do not replace deployment safety. Code initialization, database queries, migrations, and background jobs can fail even while the user-facing flag is off. Remove temporary flags after rollout so every future request does not carry old release machinery.

Rollback vs Roll Forward

Rollback is best when the old version remains compatible and switching is fast. Roll forward is safer when new data cannot be understood by old code or when a migration is already irreversible.

Define both paths before deployment:

rollback:
  stop promotion
  remove v2 from traffic
  restore v1 capacity
  verify recovery signals

roll forward:
  disable affected feature
  deploy minimal corrective change
  repair data separately if needed

The incident is not over when the old version is receiving traffic. Confirm error rate, latency, queue depth, and business outcomes return to normal.

Which Strategy to Use

Rolling:
  default for frequent service updates with strong compatibility

Blue-green:
  useful when fast whole-environment switching justifies extra capacity

Canary:
  best when production behavior is uncertain and traffic can be measured safely

Feature flag:
  useful alongside any strategy to control user-visible behavior

Teams often combine them: a rolling update introduces the new version with a feature disabled, then a canary flag gradually releases the behavior.

The Bottom Line

Deployment strategy controls how quickly a new failure reaches users and how easily traffic can move back.

The rules:

Make old and new versions coexist safely
Use readiness gates before sending traffic
Drain requests and background work before termination
Keep capacity headroom during replacement
Use expand-and-contract for data and message changes
Define canary promotion and rollback signals in advance
Test rollback against data written by the new version
Keep a bounded observation window before cleanup
Choose rolling, blue-green, or canary based on blast radius and recovery needs

Zero downtime comes from compatibility and controlled transitions. The colored boxes on the deployment diagram are the easy part.