$ cat ~/field-notes/idempotency-keys-reliable-apis.md
Idempotency Keys: Make Retries Safe Without Doing Work Twice
A client sends POST /payments. The server charges the card and begins writing the response. The connection drops before the client receives it.
Did the payment succeed?
The client cannot tell. Retrying may be necessary, but an ordinary POST retry can charge the card twice. Refusing to retry can leave a legitimate payment unfinished from the user's point of view.
This is the problem idempotency keys solve: many delivery attempts produce one logical result.
Retries Are Normal, Not an Edge Case
Requests can be duplicated by more than impatient users clicking twice:
client timeout after server completed work
browser or mobile SDK automatic retry
proxy retry after a connection failure
queue redelivery after a worker crash
two application servers handling the same event
At-least-once delivery is common. A network can usually tell that a response was not received; it cannot reliably tell whether the remote side completed the work before the failure.
An idempotent operation has the same externally visible outcome when performed repeatedly. A PUT that replaces a resource at a known URL is naturally close to idempotent. Creating an order with POST or sending money is not, unless the application supplies an identity for the intended operation.
The Client Names the Intended Operation
For each logical write, the client generates a high-entropy value and sends it with the request:
POST /v1/payments
Idempotency-Key: 01JZKXQ52A7HKPT6BF3PJ3B1FH
Content-Type: application/json
{"amount":4999,"currency":"USD","customer_id":"cus_123"}
The key belongs to the intention to create this payment, not to the HTTP attempt. If the client times out, it retries with the same key. If the user starts a separate payment later, it uses a new key.
UUIDv7, ULIDs, or another cryptographically random identifier work well. Do not derive a key from a timestamp, incrementing counter, or easily guessed order number. A key must be unique enough that unrelated operations never collide.
What the Server Stores
The server needs a durable record associated with the operation. A useful shape is:
caller_id tenant or authenticated principal
operation endpoint or operation name
idempotency_key client-provided unique value
request_fingerprint hash of the meaningful request fields
status in_progress | completed | failed
response_status original HTTP status
response_body original response, or reference to it
created_at
expires_at
The uniqueness constraint is the foundation:
CREATE UNIQUE INDEX idempotency_operations_unique
ON idempotency_operations (caller_id, operation, idempotency_key);
Scoping by caller prevents one account from replaying or colliding with another account's key. Scoping by operation makes accidental reuse across unrelated endpoints explicit.
Store a fingerprint of the canonical request alongside the key. If a later request presents the same key but a different amount or destination, return a conflict instead of silently treating two different intentions as one.
HTTP/1.1 409 Conflict
Content-Type: application/json
{"error":"idempotency_key_reused_with_different_request"}
Never use a plain unsalted hash of sensitive request data as an identifier that appears in logs. Store only what is needed, and protect the record like any other payment or user data.
The Basic Request Flow
The first request creates a claim. Later requests either receive the saved result or learn that work is still in progress.
request (key K)
|
v
atomically insert claim for K
|
+-- already completed --> return saved status and body
|
+-- already in progress -> return 409/202, or wait briefly
|
+-- newly claimed ------> perform operation
|
v
save final result for K
|
v
return that result
The exact in-progress response is a product decision. For a request that normally completes in a few hundred milliseconds, a duplicate can wait a bounded time for the original result, then receive 409 Conflict with a retry hint. For a long-running task, 202 Accepted with an operation URL may be clearer.
What matters is that duplicate work does not begin.
The Race You Must Close
This incorrect implementation looks plausible:
const existing = await findIdempotencyKey(key);
if (existing) return existing.response;
const payment = await chargeCard(request);
await saveIdempotencyKey(key, payment);
return payment;
Two requests can both run findIdempotencyKey before either saves a result. Both charge the card.
Create the claim with a database uniqueness constraint, transactional insert, conditional write, or distributed store primitive that is genuinely atomic. Do not rely on an in-process mutex: another process, deployment, or retry worker will not share it.
For work that changes local database state, the strongest design usually commits the business record and the completed idempotency record in the same database transaction:
transaction:
insert idempotency claim (must be new)
create order
reserve inventory
save successful response for the claim
commit
If the transaction rolls back, neither the order nor the completed key exists, so a retry can safely try again.
External Side Effects Need Their Own Identity
A database transaction cannot roll back an email already sent or a payment gateway charge already accepted.
The unsafe gap is:
charge payment provider
process crashes
save local idempotency result never happens
retry charges provider again
Pass the same operation identity to the downstream system when it supports idempotency. Payment providers often accept an idempotency key; use a stable, namespaced value such as payment:{our_operation_id}. If a downstream system lacks this feature, design an explicit reconciliation path using a provider-side reference, and assume an ambiguous outcome can require repair.
For asynchronous effects, use an outbox:
transaction:
create order
save idempotency result
write "order created" outbox event
commit
relay publishes outbox event with a stable event ID
consumer deduplicates by that event ID
The outbox avoids losing an event after the database commit. Consumer-side deduplication avoids applying an at-least-once event twice. Idempotency is a property of the whole path, not just the API boundary.
Cache Is Not Enough
It is tempting to put keys in Redis with a short TTL. That can be useful as a performance layer, but it is often insufficient as the source of truth.
If Redis evicts a key, restarts without persistence, or its TTL expires while clients are still retrying, a duplicate request becomes a new operation. For irreversible writes, keep the authoritative idempotency record in durable storage alongside the business data.
Choose retention from real retry behavior:
client retry deadline
+ proxy and job-delivery retry window
+ maximum delayed or manual retry policy
+ safety margin
= minimum idempotency record retention
A payment key retained for only five minutes is not safe if a mobile client can resume and retry an hour later. Expire old records deliberately, but do not delete them before the system has stopped treating a retry as plausible.
Decide Which Results Are Replayable
Save and replay a completed success. Also consider saving deterministic client errors.
For example, if the original request is invalid because amount is negative, repeating the exact request should receive the same validation response. That prevents pointless repeated processing.
Be more careful with temporary failures:
validation failure usually replay the 4xx response
business rejection usually replay the final 4xx response
successful operation replay the 2xx response
dependency timeout leave retryable or record an in-progress/unknown state
internal crash do not blindly freeze a transient 500 forever
The key question is whether the operation's outcome is known. If an external charge may have succeeded but the application timed out, the state is not a simple failure. Mark it for reconciliation or query the downstream system by its stable reference before accepting another attempt.
Make the Contract Visible to Clients
Document the behavior so SDKs and callers use it correctly:
one key per logical write
reuse the same key only when retrying that write
reuse with changed parameters returns conflict
key scope and maximum length
how long the service retains records
how duplicates behave while the first request is running
which response statuses are replayed
Log the key or a safe keyed representation with the request ID, operation ID, caller, and final state. These fields make support incidents tractable: a team can answer "was this order created?" without guessing from a timeout report.
Track duplicate hits, key conflicts, in-progress age, reconciliation outcomes, and expired-key retries. A sudden increase in duplicate hits may reveal a client timeout regression or an unhealthy proxy.
Idempotency Is Not Exactly-Once Delivery
An idempotency key does not make a distributed system exactly once. It gives one application-defined operation a durable identity and suppresses repeated handling within that contract.
There can still be duplicate emails, a key can expire, a downstream provider can have different semantics, and a crash can leave an outcome unknown. Good designs make those cases observable and repairable instead of pretending the network cannot fail.
The Bottom Line
Idempotency keys make safe retries possible when clients cannot distinguish a lost response from a lost request.
The rules:
- Generate one unpredictable key per logical write, not per HTTP attempt
- Scope the key to the authenticated caller and operation
- Atomically claim the key before executing side effects
- Store a request fingerprint and reject changed requests using the same key
- Persist the final response with the business change when possible
- Give downstream side effects their own stable idempotency or event identity
- Treat unknown external outcomes as reconciliation problems, not automatic retries
- Retain records longer than every realistic retry window
- Document and measure the duplicate and in-progress behavior
Retries are unavoidable. Doing the work twice is a design choice.