cd ..
$ cat ~/field-notes/retry-failed-requests.md
Retrying Failed Requests: Exponential Backoff Explained
Our payment webhook failed. Stripe sent the payment confirmation, our API returned 500, Stripe never retried. Customer charged but order never fulfilled. Manual reconciliation for 50 orders.
I added retry logic with exponential backoff. Transient failures now retry automatically. Haven't had a lost webhook since. Five lines of code prevented a recurring nightmare.
Here's how to retry failed requests properly, with exponential backoff, jitter, and real production patterns.
Why Requests Fail
Network requests fail all the time:
// This will fail eventually
const response = await fetch('https://api.example.com/data');
Common failures:
- Network timeout (WiFi drops, slow connection)
- Server temporarily down (restart, deploy)
- Rate limiting (429 Too Many Requests)
- Transient errors (500 Internal Server Error)
- Database connection issues (temporary)
One-time requests fail permanently. Retrying fixes most failures.
The Simplest Retry
async function fetchWithRetry(url, maxRetries = 3) {
for (let i = 0; i < maxRetries; i++) {
try {
const response = await fetch(url);
if (response.ok) {
return await response.json();
}
// Non-ok response, throw to retry
throw new Error(`HTTP ${response.status}`);
} catch (error) {
// Last attempt, throw error
if (i === maxRetries - 1) {
throw error;
}
console.log(`Attempt ${i + 1} failed, retrying...`);
}
}
}
// Usage
try {
const data = await fetchWithRetry('https://api.example.com/data', 3);
console.log(data);
} catch (error) {
console.error('All retries failed:', error);
}
Problem: Retries immediately. Hammers failing server.
Adding Delays
Wait between retries:
function sleep(ms) {
return new Promise(resolve => setTimeout(resolve, ms));
}
async function fetchWithRetry(url, maxRetries = 3, delayMs = 1000) {
for (let i = 0; i < maxRetries; i++) {
try {
const response = await fetch(url);
if (response.ok) {
return await response.json();
}
throw new Error(`HTTP ${response.status}`);
} catch (error) {
if (i === maxRetries - 1) {
throw error;
}
console.log(`Attempt ${i + 1} failed, waiting ${delayMs}ms...`);
await sleep(delayMs);
}
}
}
// Try 3 times with 1 second between attempts
const data = await fetchWithRetry('https://api.example.com/data', 3, 1000);
Problem: Fixed delay. All clients retry at same time (thundering herd).
Exponential Backoff
Double the delay after each retry:
async function fetchWithRetry(url, maxRetries = 5) {
let delay = 1000; // Start with 1 second
for (let i = 0; i < maxRetries; i++) {
try {
const response = await fetch(url);
if (response.ok) {
return await response.json();
}
throw new Error(`HTTP ${response.status}`);
} catch (error) {
if (i === maxRetries - 1) {
throw error;
}
console.log(`Attempt ${i + 1} failed, waiting ${delay}ms...`);
await sleep(delay);
delay *= 2; // Double the delay: 1s, 2s, 4s, 8s, 16s
}
}
}
// Retry with exponential backoff:
// Attempt 1: fail → wait 1s
// Attempt 2: fail → wait 2s
// Attempt 3: fail → wait 4s
// Attempt 4: fail → wait 8s
// Attempt 5: fail → throw error
Delays:
Attempt 1: 0s (immediate)
Attempt 2: 1s delay
Attempt 3: 2s delay
Attempt 4: 4s delay
Attempt 5: 8s delay
Total time: 15 seconds max
Adding Jitter
Randomize delays to prevent thundering herd:
function sleep(ms) {
return new Promise(resolve => setTimeout(resolve, ms));
}
async function fetchWithRetry(url, maxRetries = 5) {
let delay = 1000;
for (let i = 0; i < maxRetries; i++) {
try {
const response = await fetch(url);
if (response.ok) {
return await response.json();
}
throw new Error(`HTTP ${response.status}`);
} catch (error) {
if (i === maxRetries - 1) {
throw error;
}
// Add jitter: random delay between 0 and calculated delay
const jitter = Math.random() * delay;
console.log(`Attempt ${i + 1} failed, waiting ${Math.round(jitter)}ms...`);
await sleep(jitter);
delay *= 2;
}
}
}
// With jitter:
// Attempt 2: 0-1000ms (random)
// Attempt 3: 0-2000ms (random)
// Attempt 4: 0-4000ms (random)
// Attempt 5: 0-8000ms (random)
Why jitter matters:
Without jitter:
Server crashes at 12:00:00
1000 clients all retry at 12:00:01 → server crashes again
1000 clients all retry at 12:00:03 → server crashes again
With jitter:
Server crashes at 12:00:00
1000 clients retry randomly between 12:00:01-12:00:02
Server recovers gradually
Full Implementation
class RetryableError extends Error {
constructor(message, retryable = true) {
super(message);
this.retryable = retryable;
}
}
async function fetchWithRetry(url, options = {}) {
const {
maxRetries = 5,
initialDelay = 1000,
maxDelay = 32000,
factor = 2,
onRetry = null
} = options;
let delay = initialDelay;
for (let attempt = 0; attempt < maxRetries; attempt++) {
try {
const response = await fetch(url);
// Success
if (response.ok) {
return await response.json();
}
// Client errors (4xx) - don't retry
if (response.status >= 400 && response.status < 500) {
if (response.status === 429) {
// Rate limited - retry with Retry-After header
const retryAfter = response.headers.get('Retry-After');
if (retryAfter) {
delay = parseInt(retryAfter) * 1000;
}
throw new RetryableError(`Rate limited: ${response.status}`);
}
// Other 4xx - don't retry
throw new RetryableError(
`Client error: ${response.status}`,
false
);
}
// Server errors (5xx) - retry
throw new RetryableError(`Server error: ${response.status}`);
} catch (error) {
const isLastAttempt = attempt === maxRetries - 1;
// Don't retry if error is not retryable
if (error instanceof RetryableError && !error.retryable) {
throw error;
}
// Last attempt - throw error
if (isLastAttempt) {
throw error;
}
// Calculate delay with jitter
const jitter = Math.random() * delay;
const actualDelay = Math.min(jitter, maxDelay);
// Call retry callback if provided
if (onRetry) {
onRetry(attempt + 1, actualDelay, error);
}
console.log(
`Attempt ${attempt + 1}/${maxRetries} failed: ${error.message}. ` +
`Retrying in ${Math.round(actualDelay)}ms...`
);
await sleep(actualDelay);
delay = Math.min(delay * factor, maxDelay);
}
}
}
// Usage
try {
const data = await fetchWithRetry('https://api.example.com/data', {
maxRetries: 5,
initialDelay: 1000,
maxDelay: 32000,
onRetry: (attempt, delay, error) => {
console.log(`Retry attempt ${attempt} after ${delay}ms: ${error.message}`);
}
});
console.log('Success:', data);
} catch (error) {
console.error('Failed after retries:', error);
}
Retry Only Specific Errors
function isRetryable(error, response) {
// Network errors - always retry
if (error instanceof TypeError && error.message.includes('fetch')) {
return true;
}
// No response - retry
if (!response) {
return true;
}
// Rate limiting - retry
if (response.status === 429) {
return true;
}
// Server errors - retry
if (response.status >= 500) {
return true;
}
// Service unavailable - retry
if (response.status === 503) {
return true;
}
// Gateway errors - retry
if (response.status === 502 || response.status === 504) {
return true;
}
// Client errors - don't retry
if (response.status >= 400 && response.status < 500) {
return false;
}
return false;
}
async function fetchWithRetry(url, maxRetries = 5) {
let delay = 1000;
for (let attempt = 0; attempt < maxRetries; attempt++) {
try {
const response = await fetch(url);
if (response.ok) {
return await response.json();
}
// Check if retryable
if (!isRetryable(null, response)) {
throw new Error(`Non-retryable error: ${response.status}`);
}
throw new Error(`HTTP ${response.status}`);
} catch (error) {
// Network error - check if retryable
if (!isRetryable(error, null)) {
throw error;
}
if (attempt === maxRetries - 1) {
throw error;
}
const jitter = Math.random() * delay;
await sleep(jitter);
delay *= 2;
}
}
}
Respecting Retry-After Header
async function fetchWithRetry(url, maxRetries = 5) {
let delay = 1000;
for (let attempt = 0; attempt < maxRetries; attempt++) {
try {
const response = await fetch(url);
if (response.ok) {
return await response.json();
}
// Rate limited - respect Retry-After header
if (response.status === 429) {
const retryAfter = response.headers.get('Retry-After');
if (retryAfter) {
// Retry-After can be seconds or HTTP date
const retryDelay = isNaN(retryAfter)
? new Date(retryAfter).getTime() - Date.now()
: parseInt(retryAfter) * 1000;
console.log(`Rate limited, waiting ${retryDelay}ms as requested`);
await sleep(retryDelay);
continue;
}
}
throw new Error(`HTTP ${response.status}`);
} catch (error) {
if (attempt === maxRetries - 1) {
throw error;
}
const jitter = Math.random() * delay;
await sleep(jitter);
delay *= 2;
}
}
}
Axios with Retry
const axios = require('axios');
const axiosRetry = require('axios-retry');
// Configure axios to retry
axiosRetry(axios, {
retries: 5,
retryDelay: axiosRetry.exponentialDelay,
retryCondition: (error) => {
// Retry on network errors or 5xx responses
return axiosRetry.isNetworkOrIdempotentRequestError(error)
|| error.response?.status >= 500;
},
onRetry: (retryCount, error, requestConfig) => {
console.log(`Retry attempt ${retryCount} for ${requestConfig.url}`);
}
});
// Use axios normally - retries automatically
try {
const response = await axios.get('https://api.example.com/data');
console.log(response.data);
} catch (error) {
console.error('Failed after retries:', error);
}
Promise-Retry Library
const promiseRetry = require('promise-retry');
const data = await promiseRetry(async (retry, number) => {
console.log(`Attempt ${number}`);
try {
const response = await fetch('https://api.example.com/data');
if (!response.ok) {
throw new Error(`HTTP ${response.status}`);
}
return await response.json();
} catch (error) {
// Retry on failure
retry(error);
}
}, {
retries: 5,
factor: 2,
minTimeout: 1000,
maxTimeout: 32000,
randomize: true // Add jitter
});
Webhook Retry Pattern
class WebhookRetryQueue {
constructor() {
this.queue = [];
this.processing = false;
}
async send(url, payload) {
this.queue.push({ url, payload, attempts: 0 });
if (!this.processing) {
this.processQueue();
}
}
async processQueue() {
this.processing = true;
while (this.queue.length > 0) {
const webhook = this.queue[0];
try {
await this.sendWithRetry(webhook.url, webhook.payload);
this.queue.shift(); // Success - remove from queue
} catch (error) {
webhook.attempts++;
if (webhook.attempts >= 5) {
console.error('Webhook failed after 5 attempts:', webhook.url);
this.queue.shift(); // Give up
} else {
// Move to back of queue
this.queue.shift();
this.queue.push(webhook);
// Wait before processing next
await sleep(Math.pow(2, webhook.attempts) * 1000);
}
}
}
this.processing = false;
}
async sendWithRetry(url, payload) {
const response = await fetch(url, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(payload)
});
if (!response.ok) {
throw new Error(`HTTP ${response.status}`);
}
return response;
}
}
// Usage
const webhookQueue = new WebhookRetryQueue();
app.post('/order', async (req, res) => {
const order = await createOrder(req.body);
// Send webhook - will retry on failure
webhookQueue.send('https://partner.com/webhook', {
event: 'order.created',
order: order
});
res.json(order);
});
Circuit Breaker Pattern
Stop retrying if service is down:
class CircuitBreaker {
constructor(threshold = 5, timeout = 60000) {
this.failureCount = 0;
this.threshold = threshold;
this.timeout = timeout;
this.state = 'CLOSED'; // CLOSED, OPEN, HALF_OPEN
this.nextAttempt = Date.now();
}
async execute(fn) {
if (this.state === 'OPEN') {
if (Date.now() < this.nextAttempt) {
throw new Error('Circuit breaker is OPEN');
}
this.state = 'HALF_OPEN';
}
try {
const result = await fn();
this.onSuccess();
return result;
} catch (error) {
this.onFailure();
throw error;
}
}
onSuccess() {
this.failureCount = 0;
this.state = 'CLOSED';
}
onFailure() {
this.failureCount++;
if (this.failureCount >= this.threshold) {
this.state = 'OPEN';
this.nextAttempt = Date.now() + this.timeout;
console.log(`Circuit breaker OPEN for ${this.timeout}ms`);
}
}
}
// Usage
const breaker = new CircuitBreaker(5, 60000); // Open after 5 failures, retry after 60s
async function callAPI() {
return await breaker.execute(async () => {
return await fetchWithRetry('https://api.example.com/data');
});
}
Real-World Example
class APIClient {
constructor(baseURL) {
this.baseURL = baseURL;
}
async request(endpoint, options = {}) {
const {
method = 'GET',
body = null,
maxRetries = 5
} = options;
let delay = 1000;
for (let attempt = 0; attempt < maxRetries; attempt++) {
try {
const response = await fetch(`${this.baseURL}${endpoint}`, {
method,
headers: {
'Content-Type': 'application/json',
'Authorization': `Bearer ${this.getToken()}`
},
body: body ? JSON.stringify(body) : null
});
// Success
if (response.ok) {
return await response.json();
}
// Rate limited
if (response.status === 429) {
const retryAfter = response.headers.get('Retry-After');
delay = retryAfter ? parseInt(retryAfter) * 1000 : delay;
throw new Error('Rate limited');
}
// Client error - don't retry
if (response.status >= 400 && response.status < 500) {
throw new Error(`Client error: ${response.status}`);
}
// Server error - retry
throw new Error(`Server error: ${response.status}`);
} catch (error) {
// Don't retry client errors
if (error.message.includes('Client error')) {
throw error;
}
const isLastAttempt = attempt === maxRetries - 1;
if (isLastAttempt) {
throw new Error(`Failed after ${maxRetries} attempts: ${error.message}`);
}
const jitter = Math.random() * delay;
console.log(`Attempt ${attempt + 1} failed, retrying in ${Math.round(jitter)}ms...`);
await sleep(jitter);
delay = Math.min(delay * 2, 32000);
}
}
}
getToken() {
return 'your-api-token';
}
}
// Usage
const api = new APIClient('https://api.example.com');
try {
const data = await api.request('/users', { maxRetries: 3 });
console.log(data);
} catch (error) {
console.error('API request failed:', error);
}
Testing Retry Logic
describe('fetchWithRetry', () => {
it('should succeed on first attempt', async () => {
const mockFetch = jest.fn().mockResolvedValue({
ok: true,
json: async () => ({ data: 'success' })
});
global.fetch = mockFetch;
const result = await fetchWithRetry('https://api.example.com/data');
expect(result).toEqual({ data: 'success' });
expect(mockFetch).toHaveBeenCalledTimes(1);
});
it('should retry on failure and eventually succeed', async () => {
const mockFetch = jest.fn()
.mockRejectedValueOnce(new Error('Network error'))
.mockRejectedValueOnce(new Error('Network error'))
.mockResolvedValueOnce({
ok: true,
json: async () => ({ data: 'success' })
});
global.fetch = mockFetch;
const result = await fetchWithRetry('https://api.example.com/data', 3);
expect(result).toEqual({ data: 'success' });
expect(mockFetch).toHaveBeenCalledTimes(3);
});
it('should throw after max retries', async () => {
const mockFetch = jest.fn().mockRejectedValue(new Error('Network error'));
global.fetch = mockFetch;
await expect(
fetchWithRetry('https://api.example.com/data', 3)
).rejects.toThrow('Network error');
expect(mockFetch).toHaveBeenCalledTimes(3);
});
});
Common Mistakes
Mistake 1: Retrying Non-Idempotent Operations
// BAD - Retrying POST can create duplicates
async function createUser(userData) {
return await fetchWithRetry('/users', {
method: 'POST',
body: userData
});
}
// Request fails after creating user → retry creates duplicate!
// GOOD - Use idempotency key
async function createUser(userData) {
return await fetch('/users', {
method: 'POST',
headers: {
'Idempotency-Key': generateUniqueKey()
},
body: userData
});
}
Mistake 2: No Maximum Delay
// BAD - Delay grows forever
let delay = 1000;
delay *= 2; // 1s, 2s, 4s, 8s, 16s, 32s, 64s, 128s...
// GOOD - Cap maximum delay
const maxDelay = 32000;
delay = Math.min(delay * 2, maxDelay);
Mistake 3: Not Logging Retries
// BAD - Silent retries
await fetchWithRetry(url);
// GOOD - Log retries for debugging
await fetchWithRetry(url, {
onRetry: (attempt, delay, error) => {
console.log(`Retry ${attempt} after ${delay}ms: ${error.message}`);
}
});
The Bottom Line
Network requests fail. Retrying with exponential backoff handles transient failures gracefully.
Start with 1 second, double each retry - 1s, 2s, 4s, 8s prevents hammering.
Add jitter - randomize delays to prevent thundering herd problems.
Respect Retry-After - honor rate limit headers from servers.
Don't retry forever - max 5 retries is usually enough.
We lost payment webhooks because we didn't retry. Added exponential backoff, haven't lost one since. Five lines of code fixed a recurring problem.
Add retry logic to critical requests today. Start simple with exponential backoff. Your users won't notice failures anymore.