cd ..

$ cat ~/field-notes/distributed-locks-fencing-tokens.md

Distributed Locks: Why the Lease Is Only Half the Story

Two workers must not generate the same monthly invoice. The first worker acquires a distributed lock, starts reading, and pauses long enough for its lease to expire. The second acquires the lock and finishes the invoice. Then the first resumes.

Both workers believe they have work to do. Only one currently owns the lease, but that does not stop the other from sending a write.

A Timeout Does Not Stop a Process

Lease expiry lets a system recover when an owner disappears. It cannot distinguish a dead owner from a paused owner that may return later. A long runtime pause, a machine suspension, or delayed network traffic can outlast the lease.

Checking ownership immediately before a write still leaves a gap between the check and the write. Martin Kleppmann's distributed locking analysis explains why that gap matters for correctness.

For an invoice system, "probably only one worker" is not a sufficient invariant. The durable destination must enforce what duplicate or stale work means.

Fence the Destination

A fencing token is an increasing ownership generation. Every newly granted lock receives a token larger than the previous grant. The protected resource records the largest token it has accepted and rejects older ones. Hazelcast's FencedLock documentation describes this resource-side enforcement.

worker A receives token 71 and pauses
worker B receives token 72 and writes successfully
worker A resumes and sends token 71
destination rejects A's stale write

A random owner ID is useful for identifying a lease. It does not establish which owner is newer. A timestamp from each worker's local clock is not a reliable replacement for an ordered grant protocol either.

The lock service and destination have to agree on the contract. Attaching a token to a log line provides visibility, but it does not fence anything.

An Atomic Conditional Write

Suppose a document publisher stores its current accepted generation with its output. A single-write-per-grant design could use this conditional update:

UPDATE generated_documents
SET contents = :contents,
    last_fence = :new_fence
WHERE document_id = :document_id
  AND last_fence < :new_fence
RETURNING document_id;

Assume the row already exists with a non-null initial generation. Zero returned rows means the writer cannot publish; it must stop or reconcile the already-completed result. Retrying the same grant also returns no row, so the caller needs a way to distinguish a prior success from a superseded attempt.

This is an illustrative protocol, not a complete lock implementation. Multiple writes under one grant need a more detailed ownership and operation contract. Most importantly, the comparison and write must happen atomically at the destination.

Fencing orders accepted writes. It does not magically stop an expired worker before a newer generation has reached the destination. If publication must prove current ownership at that instant, make ownership validation part of the same authoritative transaction.

Prefer the Invariant You Can State in SQL

For the original monthly invoice, a unique business key may express the actual requirement directly:

CREATE UNIQUE INDEX invoices_one_per_period
ON invoices (customer_id, billing_period);

Make those columns non-null. Then create the invoice and its related ledger changes in one transaction. A second worker can discover the existing result rather than racing to publish another invoice.

This approach still needs a plan for external effects. A uniqueness constraint cannot unsend an email or undo a remote charge. Record the intended effect through an outbox and give the receiving system a stable idempotency key.

Decide Whether the Lock Protects Efficiency or Correctness

Duplicate cache refreshes may merely waste resources. Duplicate invoices can change money owed. Use that difference to choose the mechanism and its failure policy.

For any correctness-critical lock, review one concrete question: what happens if an old owner resumes after another owner has finished? Write down the destination's rejection rule, how retries discover an existing result, and how operators repair uncertain external outcomes.

A lease helps coordinate who should work. Your durable invariant decides whose result can become real.