Webhooks: How to Build Reliable Event Delivery

Stripe's webhook system processes hundreds of millions of events per day with at-least-once delivery guarantees. I built a webhook system that lost events every time a downstream service was slow. Here's what I learned by getting it wrong first.

What Webhooks Actually Are

A webhook is an HTTP POST request your server sends to someone else's server when something happens:

Event happens in your system
        ↓
POST https://customer-app.com/webhooks/your-service
Content-Type: application/json

{
  "event": "payment.completed",
  "data": { "amount": 1000, "currency": "usd", "order_id": "ord_123" },
  "timestamp": "2026-02-16T10:30:00Z",
  "id": "evt_abc123"
}

The receiver processes it and returns 200. Simple in theory, tricky in production.

Sending Webhooks: The Basics

// BAD - synchronous, no retry, no signature
async function processPayment(order) {
    await chargeCard(order);

    // If this fails, customer never gets notified
    await fetch(order.webhookUrl, {
        method: 'POST',
        body: JSON.stringify({ event: 'payment.completed', orderId: order.id }),
    });
}

// GOOD - async with queue
async function processPayment(order) {
    const result = await chargeCard(order);

    // Queue webhook delivery, don't block payment flow
    await webhookQueue.add({
        url: order.webhookUrl,
        payload: {
            id: generateEventId(),
            event: 'payment.completed',
            data: { orderId: order.id, amount: result.amount },
            timestamp: new Date().toISOString(),
        }
    });

    return result;
}

Signature Verification: Don't Skip This

Without signatures, anyone can send fake webhooks to your customers:

// BAD - no verification, accept anything
app.post('/webhooks/payment', async (req, res) => {
    const { event, data } = req.body;
    await handleEvent(event, data);
    res.sendStatus(200);
});

// GOOD - verify HMAC signature (how Stripe does it)
const crypto = require('crypto');

// Sender: sign the webhook
function signWebhook(payload, secret) {
    const timestamp = Math.floor(Date.now() / 1000);
    const signedPayload = `${timestamp}.${JSON.stringify(payload)}`;
    const signature = crypto
        .createHmac('sha256', secret)
        .update(signedPayload)
        .digest('hex');

    return {
        'X-Webhook-Timestamp': timestamp.toString(),
        'X-Webhook-Signature': `sha256=${signature}`,
    };
}

// Receiver: verify the webhook
function verifyWebhook(req, secret) {
    const timestamp = req.headers['x-webhook-timestamp'];
    const signature = req.headers['x-webhook-signature'];

    // Prevent replay attacks: reject webhooks older than 5 minutes
    const age = Math.floor(Date.now() / 1000) - parseInt(timestamp);
    if (age > 300) {
        throw new Error('Webhook too old');
    }

    const signedPayload = `${timestamp}.${JSON.stringify(req.body)}`;
    const expected = 'sha256=' + crypto
        .createHmac('sha256', secret)
        .update(signedPayload)
        .digest('hex');

    if (!crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expected))) {
        throw new Error('Invalid signature');
    }
}

// Important: use express.raw(), NOT express.json()
// Parsing changes the body, breaking signature verification
app.post('/webhooks/payment', express.raw({ type: 'application/json' }), async (req, res) => {
    try {
        verifyWebhook(req, process.env.WEBHOOK_SECRET);
    } catch (e) {
        return res.status(400).send('Invalid signature');
    }

    const payload = JSON.parse(req.body);
    await handleEvent(payload.event, payload.data);
    res.sendStatus(200);
});

Retry Logic with Exponential Backoff

Receivers go down. You need retries:

// Webhook delivery worker
async function deliverWebhook(job) {
    const { url, payload, attempt = 1 } = job.data;

    try {
        const response = await fetch(url, {
            method: 'POST',
            headers: {
                'Content-Type': 'application/json',
                'X-Webhook-ID': payload.id,
                'X-Webhook-Attempt': attempt.toString(),
                ...signWebhook(payload, getSecretForEndpoint(url)),
            },
            body: JSON.stringify(payload),
            signal: AbortSignal.timeout(10000),  // 10s timeout
        });

        if (response.ok) {
            await logDelivery(payload.id, url, 'success', response.status);
            return;
        }

        throw new Error(`HTTP ${response.status}`);

    } catch (error) {
        await logDelivery(payload.id, url, 'failed', null, error.message);

        if (attempt >= 10) {
            await markWebhookFailed(payload.id, url, error.message);
            return;
        }

        // Exponential backoff: 1s, 2s, 4s, 8s... up to 1 hour
        const delay = Math.min(1000 * Math.pow(2, attempt - 1), 3600000);

        await webhookQueue.add(
            { url, payload, attempt: attempt + 1 },
            { delay }
        );
    }
}

Retry schedule:

Attempt 1: immediate
Attempt 2: 1 second
Attempt 3: 2 seconds
Attempt 4: 4 seconds
Attempt 5: 8 seconds
Attempt 6: 16 seconds
Attempt 7: 32 seconds
Attempt 8: 1 minute
Attempt 9: 2 minutes
Attempt 10: 4 minutes
→ Give up, alert customer

Handling Webhooks: Return 200 Fast

// BAD - slow processing blocks webhook, sender retries
app.post('/webhooks/stripe', async (req, res) => {
    verifyStripeSignature(req);

    const { type, data } = req.body;

    // This might take 5+ seconds
    await fulfillOrder(data.object.metadata.orderId);
    await sendConfirmationEmail(data.object.customer_email);
    await updateAnalytics(data.object);

    res.sendStatus(200);  // If this takes > 30s, Stripe retries
});

// GOOD - acknowledge immediately, process async
app.post('/webhooks/stripe', async (req, res) => {
    verifyStripeSignature(req);

    // Queue for background processing
    await eventQueue.add({
        source: 'stripe',
        type: req.body.type,
        data: req.body.data,
        receivedAt: new Date().toISOString(),
    });

    res.sendStatus(200);  // Returns in < 100ms
});

Idempotency: Handle Duplicate Deliveries

Webhooks are delivered at-least-once. You will get duplicates. Handle them:

// BAD - processing same event twice charges customer twice
app.post('/webhooks/payment', async (req, res) => {
    if (req.body.event === 'payment.completed') {
        await fulfillOrder(req.body.data.orderId);  // Runs twice!
    }
    res.sendStatus(200);
});

// GOOD - idempotent with deduplication
app.post('/webhooks/payment', async (req, res) => {
    const eventId = req.body.id;

    // Check if we've seen this event
    const existing = await db.query(
        'SELECT id FROM processed_events WHERE event_id = $1',
        [eventId]
    );

    if (existing.rows.length > 0) {
        return res.sendStatus(200);  // Already processed
    }

    // Process and record atomically
    await db.transaction(async (trx) => {
        // Unique constraint prevents duplicate processing
        await trx.query(
            'INSERT INTO processed_events (event_id, processed_at) VALUES ($1, NOW())',
            [eventId]
        );

        await fulfillOrder(req.body.data.orderId, trx);
    });

    res.sendStatus(200);
});

Delivery Log for Debugging

When a customer says "I didn't get the event," you need a record:

CREATE TABLE webhook_deliveries (
    id SERIAL PRIMARY KEY,
    webhook_id TEXT NOT NULL,
    endpoint_url TEXT NOT NULL,
    event_type TEXT NOT NULL,
    payload JSONB NOT NULL,
    attempt INTEGER NOT NULL DEFAULT 1,
    status TEXT NOT NULL,  -- 'success', 'failed', 'pending'
    response_code INTEGER,
    error_message TEXT,
    duration_ms INTEGER,
    attempted_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX idx_webhook_deliveries_webhook_id ON webhook_deliveries(webhook_id);
CREATE INDEX idx_webhook_deliveries_endpoint ON webhook_deliveries(endpoint_url, attempted_at DESC);

Now you can answer "did you try to deliver it?" with a query instead of "check your logs."

Common Mistakes

Not verifying signatures

Any server on the internet can POST to your webhook endpoint. Without signature verification, attackers can trigger arbitrary events in your system. Always verify.

Using the same secret for all customers

// BAD
const secret = process.env.WEBHOOK_SECRET;

// GOOD - per-customer secrets
const secret = await getCustomerWebhookSecret(customerId);
// Compromise of one customer's secret doesn't affect others

Accepting huge payloads without limits

// Set a payload size limit
app.post('/webhooks', express.raw({
    type: 'application/json',
    limit: '1mb',  // Reject oversized requests
}), handler);

The Bottom Line

Webhooks are simple to implement badly. The reliability is in the details.

The checklist:

• Sign with HMAC-SHA256, verify on receipt
• Return 200 immediately, process async
• Retry with exponential backoff (at least 10 attempts)
• Make event processing idempotent
• Log every delivery attempt
• Reject old webhooks (>5 min) to prevent replay attacks
• Set timeouts on outgoing requests

Build it right once and you won't have to explain to customers why their order confirmation never arrived.

---