cd ..

$ cat ~/field-notes/circuit-breakers-explained.md

Circuit Breakers: Stop Waiting on a Dependency That Is Down

Your product page calls a recommendations service. That service slows down, so every product page holds a connection open waiting for it. Soon the product service is exhausted too.

Recommendations were optional. Their failure became mandatory for everyone.

A circuit breaker can limit that spread by deciding when another remote attempt is unlikely to help.

Three States, One Recovery Path

The usual state machine has three states:

closed     → allow calls and observe results
open       → reject calls without contacting the dependency
half-open  → allow a bounded number of recovery probes

Sustained failure opens the circuit. After a cooldown, limited probes test recovery. Successful probes close it; failed probes reopen it. The Azure Architecture Center pattern describes this model.

Half-open is essential. Allowing all waiting callers through at the end of a timer can recreate the overload before the dependency has recovered.

Choose What Counts as Failure

A missing product is different from an unavailable product service. Counting every 404 as dependency failure can open the circuit because customers requested nonexistent IDs.

For a hypothetical recommendation client, a policy might be:

Outcome Treatment
connection failure or deadline exceeded count as dependency failure
selected 5xx responses count as dependency failure
caller validation error return normally, do not trip the circuit
429 apply the documented rate-limit policy and scope
successful response record success

This is a proposed application policy, not a universal status-code rule. An authentication failure caused by a broken service credential may require an immediate alert rather than repeated recovery probes.

Use a minimum sample size before applying a failure percentage. One failed call out of one should not necessarily disable a low-traffic feature for minutes.

Keep the Controls Separate

A timeout limits how long one call waits. A concurrency limit caps the resources simultaneous calls can consume. A circuit breaker reduces new attempts while recent evidence says a dependency is unhealthy.

Use all three where the failure mode warrants them. The breaker cannot rescue the first wave of calls that already consumed every socket.

Retries need the same discipline. They can amplify load, and breakers add state changes that need testing. Amazon's Builders' Library discussion of retries explains both concerns. A retry loop should never bypass an open circuit or create an unlimited stream of half-open probes.

For a client wrapper, the intended structure might be:

check remaining request deadline
ask breaker for permission, including a bounded probe permit
acquire a bounded concurrency slot
call dependency with a timeout
classify the result and report it to the breaker
release permits in every completion path

Do not count a local concurrency rejection as proof that the remote service failed. Those are separate signals and may require different responses.

Scope the Circuit to the Failure

One failing optional endpoint should not disable every endpoint on the same hostname without a reason. Equally, a breaker per individual URL can create millions of independent circuits that never collect enough evidence.

Choose a bounded grouping such as dependency and operation class. Be careful with tenant-specific errors: one rate-limited customer should not necessarily cut off every other customer.

Breakers held in process memory also have separate histories on each application instance. Account for aggregate probe traffic when many instances recover together. Jittered recovery intervals can help avoid synchronized attempts.

Return a Result You Can Explain

For recommendations, a fallback might be an empty section or a recently cached list. For payment authorization, pretending success would create a different incident. Return an explicit unavailable or pending state when the business operation cannot safely continue.

Keep fallback use visible in metrics. A page that returns 200 with optional features disabled is available, but the dependency is still broken.

Exercise opening, recovery, flapping, and slow responses before relying on the breaker. Track actual dependency calls, rejected calls, state transitions, probe outcomes, and time spent degraded. The goal is to preserve useful service while giving a failing dependency room to recover.