$ cat ~/field-notes/backpressure-explained.md
Backpressure: How Fast Systems Avoid Drowning in Work
Your API receives 2,000 jobs per second. Workers complete 1,500 per second. The queue library is "handling" the burst, so nothing fails.
Every second, 500 more jobs wait. After ten minutes, 300,000 jobs are queued. Memory climbs, job latency reaches minutes, retries add traffic, and the process dies.
The queue did not absorb the overload. It hid it.
The Conservation Law
If work arrives faster than it leaves, only a few things can happen:
arrival rate > service rate
queue grows
or producer slows
or work is rejected/dropped
or capacity increases
There is no fifth option. An unbounded queue chooses "grow until something else fails."
Backpressure is how a saturated consumer communicates that condition upstream.
Bounded Queues Make Failure Explicit
class WorkQueue {
constructor({ maxQueued = 1000, concurrency = 20 }) {
this.maxQueued = maxQueued;
this.concurrency = concurrency;
this.queue = [];
this.active = 0;
}
add(task) {
if (this.queue.length >= this.maxQueued) {
throw new Error('queue_full');
}
return new Promise((resolve, reject) => {
this.queue.push({ task, resolve, reject, enqueuedAt: Date.now() });
this.drain();
});
}
drain() {
while (this.active < this.concurrency && this.queue.length > 0) {
const item = this.queue.shift();
this.active++;
Promise.resolve()
.then(item.task)
.then(item.resolve, item.reject)
.finally(() => {
this.active--;
this.drain();
});
}
}
}
This is an illustration, not a production queue. The important properties are a concurrency cap and a queue cap. Once full, the caller gets a decision instead of an eventual out-of-memory crash.
The cap should reflect how long work remains useful. If users abandon requests after two seconds, holding twenty minutes of backlog is worse than rejecting new work immediately.
Concurrency Is Not Throughput
Increasing concurrency helps only while the bottleneck has unused capacity.
10 workers → 800 jobs/s
20 workers → 1,400 jobs/s
40 workers → 1,520 jobs/s
80 workers → 1,300 jobs/s + database timeouts
Past saturation, more concurrency adds context switching, lock contention, memory, connection pressure, and retries. Find the throughput knee with load testing, then cap concurrency near it.
Different work needs different limits. A CPU-heavy image transform might use roughly one worker per core. An I/O-heavy API call may tolerate more concurrency but must stay under downstream rate and connection limits.
Streams Already Have Backpressure—If You Listen
Node.js writable streams report when their buffer is full:
import { once } from 'node:events';
async function writeRows(output, rows) {
for (const row of rows) {
const chunk = JSON.stringify(row) + '\n';
if (!output.write(chunk)) {
// Stop producing until the consumer drains its buffer.
await once(output, 'drain');
}
}
output.end();
}
Ignoring the false return value does not make writes faster. It moves every unwritten chunk into memory.
Prefer pipeline helpers that propagate errors and backpressure automatically:
import { pipeline } from 'node:stream/promises';
await pipeline(databaseRowStream, transformStream, response);
Bounded Channels in Go
jobs := make(chan Job, 100) // bounded queue
for i := 0; i < 20; i++ {
go func() {
for job := range jobs {
process(job)
}
}()
}
func submit(ctx context.Context, job Job) error {
select {
case jobs <- job:
return nil
case <-ctx.Done():
return ctx.Err()
default:
return ErrQueueFull
}
}
The default branch makes admission non-blocking. Remove it if producers should wait until capacity becomes available, but always keep the caller's cancellation path.
HTTP Admission Control
When synchronous capacity is exhausted, fail quickly and honestly.
HTTP/1.1 503 Service Unavailable
Retry-After: 2
Content-Type: application/json
{"error":"temporarily_overloaded"}
Use 429 Too Many Requests when a client, tenant, or API quota is the constraint. Use 503 when the service as a whole lacks temporary capacity.
Only send Retry-After when retrying later is reasonable. Clients should add jitter and obey an overall retry limit. Immediate synchronized retries turn one overload into another.
For asynchronous work, accept with 202 only after the job is durably stored. Returning 202 and then dropping an in-memory queue on restart is a lie.
Shed Work by Priority
Not all work deserves equal protection during overload.
highest: authentication, writes, payment confirmation
medium: normal interactive reads
lowest: analytics, prefetch, thumbnails, cache warming
Separate queues and concurrency budgets prevent low-priority batch jobs from occupying every database connection needed by user traffic.
Load shedding can also reject stale work. A notification queued for an hour may no longer be useful. Include an expiry or deadline with the job and discard it before expensive processing.
Push Back Across Service Boundaries
A bounded queue in service C helps only if services B and A respond appropriately.
client → A → B → C → database
If C returns overload, B should not retry indefinitely while holding its own request open. B should honor its deadline and propagate a bounded failure. A may serve a degraded response or reject early.
Backpressure that stops at one layer becomes queue growth at the layer before it.
Message brokers need the same controls: consumer prefetch limits, bounded in-flight messages, retry delays, dead-letter policy, and producer quotas. A durable broker protects data; it does not create infinite processing capacity.
Measure Queue Age, Not Just Depth
A depth of 1,000 could mean one second or one hour of work.
Track:
arrival rate
completion rate
active worker count
queue depth
oldest queued item age
enqueue-to-start latency
processing duration
rejection/drop count by reason and priority
downstream saturation (CPU, connections, rate limits)
Oldest-item age is often the clearest user-facing signal. Alert before it exceeds the operation's usefulness or service objective.
Scaling Can Move the Bottleneck
Adding workers increases throughput only if the downstream system can handle the extra concurrency. Autoscaling workers from 20 to 200 against a database capped at 50 connections creates a connection storm.
Scale on a combination of queue age, throughput, and downstream headroom. Put a maximum on autoscaling that respects database, third-party, and network capacity.
The Bottom Line
Backpressure turns overload from an accidental crash into an explicit control decision.
The rules:
- Bound every in-memory queue and every concurrency pool
- Stop producing when streams report a full buffer
- Reject quickly when work cannot finish within its useful deadline
- Separate high- and low-priority workloads
- Propagate overload and cancellation upstream
- Retry with limits, delay, and jitter
- Track oldest-item age as well as queue depth
- Scale only while the real bottleneck has headroom
- Accept asynchronous jobs only after durable enqueue
A healthy system does not promise to process infinite work. It defines exactly what happens when capacity is full.