cd ..

$ cat ~/field-notes/transactional-outbox-reliable-events.md

The Transactional Outbox: Publish Events Without Losing Them

7 min read 0x.run

An order is created successfully. The application commits it to the database, then publishes order.created to a message broker.

The process crashes between those two lines.

The order exists. Inventory, email, analytics, and downstream services never hear about it. Retrying the HTTP request is not a fix: the client may already have received a successful response.

This is the dual-write problem. The transactional outbox is a small pattern that turns it from a silent data-loss risk into an explicit, recoverable workflow.

Why Two Successful Calls Are Not One Transaction

The tempting implementation is simple:

const order = await database.transaction(async (tx) => {
    return tx.orders.insert(input);
});

await broker.publish('order.created', order);

There are two bad failure windows:

database commit succeeds
process crashes
message is never published

or:

broker accepts message
process crashes before local state is recorded as published
retry publishes message again

Reversing the calls only reverses the failure. Publishing first can announce an order that later fails to commit.

Distributed transactions can coordinate some databases and brokers, but they add operational complexity, do not cover every downstream side effect, and are rarely the best default for an application service. The more useful goal is usually: do not lose an event, and make duplicates safe.

Put the Event in the Same Transaction

An outbox is a table in the same durable database as the business data. The request writes the business change and an unpublished event in one local transaction.

CREATE TABLE outbox_events (
    id            uuid PRIMARY KEY,
    topic         text NOT NULL,
    aggregate_id  uuid NOT NULL,
    payload       jsonb NOT NULL,
    created_at    timestamptz NOT NULL DEFAULT now(),
    published_at  timestamptz,
    attempts      integer NOT NULL DEFAULT 0,
    last_error    text
);

CREATE INDEX outbox_events_unpublished
ON outbox_events (created_at)
WHERE published_at IS NULL;

BEGIN;

INSERT INTO orders (id, customer_id, status, total_cents)
VALUES (:order_id, :customer_id, 'confirmed', :total_cents);

INSERT INTO outbox_events (id, topic, aggregate_id, payload)
VALUES (
  :event_id,
  'order.created',
  :order_id,
  jsonb_build_object('order_id', :order_id, 'customer_id', :customer_id)
);

COMMIT;

After the commit, there are only two useful states:

transaction failed     → neither order nor event exists
transaction succeeded  → both order and event exist

The event may wait briefly before it is delivered, but it is no longer lost because a web process crashed at the wrong moment.

A Relay Delivers the Outbox

A separate worker repeatedly finds unpublished rows, sends them to the broker, and records delivery after the broker acknowledges the send.

while true:
  claim a small batch of unpublished events
  for each event:
    publish(topic, payload, event_id=event.id)
    after broker acknowledgement, mark event published

For a relational database, multiple relay workers can claim rows without doing the same work at once:

SELECT id, topic, payload
FROM outbox_events
WHERE published_at IS NULL
ORDER BY created_at
FOR UPDATE SKIP LOCKED
LIMIT 100;

SKIP LOCKED lets another worker take a different batch instead of waiting. Keep the transaction that claims work short: do not hold database locks while a slow broker call runs. Common designs use a lease column, or record an attempt and let a timed-out lease become eligible again.

The relay must be restartable. A restart should discover unfinished rows and continue, not depend on an in-memory list of messages.

Expect At-Least-Once Delivery

There is still one unavoidable ambiguous gap:

broker accepts event E
relay crashes before setting E.published_at
relay restarts and publishes E again

Trying to remove that gap usually moves it elsewhere. Instead, publish a stable event ID and make consumers idempotent.

{
  "event_id": "019a8f1e-6c5a-7d4a-a3d9-cc06d63e4de2",
  "event_type": "order.created",
  "occurred_at": "2026-07-23T14:02:19Z",
  "data": { "order_id": "..." }
}

Each consumer stores processed IDs together with its own state change:

BEGIN;

INSERT INTO processed_events (consumer_name, event_id)
VALUES ('inventory', :event_id)
ON CONFLICT DO NOTHING;

-- Only reserve inventory if the insert above created a row.

COMMIT;

For an email provider, use event_id as the provider's idempotency key when available. For a consumer that updates a projection, make the update naturally repeatable. The event ID is the identity that joins all these retries.

The outbox gives reliable at-least-once publication. End-to-end exactly once is a much stronger claim, especially once an event causes email, payments, or other external effects.

Event Shape Is a Contract

Do not serialize an internal ORM object and call it an event. Consumers need a deliberate contract that can survive independent deployments.

Include at least:

event_id       stable identifier for deduplication
event_type     named business fact, such as order.created
event_version  schema version when evolution needs it
occurred_at    when the fact happened
aggregate_id   entity the event concerns
data           the intentionally public fields

Prefer facts in the past tense: order.created, payment.captured, address.changed. A fact says what happened; it does not tell every consumer how to implement its work.

Add fields compatibly. Consumers should ignore unknown fields, and producers should avoid making a new field mandatory until old consumers can handle it. If a meaning changes rather than merely expanding, publish a new event version or type.

Ordering Is Narrower Than It Looks

An order.created event should generally arrive before order.cancelled for that same order. Global order across every order is expensive and often unnecessary.

Use the aggregate ID as a partition or routing key when the broker supports it. That preserves useful per-entity ordering while allowing unrelated orders to process in parallel.

The database query order alone does not guarantee broker order. Multiple relay workers, retries, partitions, and consumer concurrency can all reorder events. If a consumer requires a sequence, include a version number and handle an older event deliberately:

event version <= projection version  → ignore as duplicate/old
event version = projection version+1 → apply
event version > projection version+1 → wait, repair, or reload source of truth

Do not turn every event stream into a globally serialized queue merely to avoid stating the ordering requirement.

Retry Without Creating an Outage Loop

Most relay failures are temporary: a broker deployment, network timeout, or rate limit. Retry them with exponential backoff and jitter, and retain the error and attempt count for inspection.

attempt 1: retry after a short delay
attempt 2: retry later with jitter
repeated failures: alert and investigate
poison event: quarantine with enough context to repair safely

Do not mark an event published because it has failed too many times. That converts an operational alert into quiet data loss. A dead-letter or quarantine state is useful only when someone can see it, diagnose it, correct the cause, and replay it with the same event identity.

Keep the payload self-contained enough for the consumer's immediate decision. A consumer that fetches the order after every event must also handle deletion, lag, and a source that has since changed. For large or sensitive data, store a stable reference and decide explicitly how long the referenced data remains available.

Operating the Outbox

The outbox is production infrastructure. Monitor it like a queue:

unpublished event count
age of oldest unpublished event
publish success and failure rate
attempt count and lease timeouts
events by topic
consumer deduplication rate and processing lag
quarantined events

Oldest unpublished age is usually more meaningful than row count. One hundred delayed receipt emails may be tolerable; a two-minute delay in payment-confirmation events may not be.

Published rows need retention too. Keep them long enough for debugging, replay policy, and audit needs, then archive or delete them in bounded batches. Do not let a cleanup job remove the only record of an event before the business has defined its recovery window.

When an Outbox Is the Right Tool

Use an outbox when one service commits durable state and other systems must reliably learn about that change:

order created       → reserve inventory, send receipt, update analytics
user changed email  → update search index, notify security system
payment captured    → fulfill order, update ledger, notify customer

It is not necessary for an operation with no durable state change, and it is not a substitute for a workflow engine when a long-running process needs compensations, approvals, or human intervention. It is a reliability boundary between a local transaction and asynchronous work.

The Bottom Line

The transactional outbox makes a committed business change and its notification one local database decision. A relay can then deliver the notification as often as necessary until it succeeds.

The rules:

  • Write the business record and outbox event in the same database transaction
  • Use a durable relay that can resume after a crash
  • Mark delivery only after the broker acknowledges publication
  • Give every event a stable ID and make consumers idempotent
  • State ordering requirements per aggregate instead of assuming global order
  • Evolve event schemas compatibly and treat them as public contracts
  • Monitor event age, failures, retries, and quarantined work
  • Retain and clean up events according to an explicit recovery policy

The outbox does not make failures disappear. It makes the important failure—"the database changed but nobody was told"—durable, visible, and recoverable.