cd ..

$ cat ~/field-notes/file-uploads-done-right.md

File Uploads Done Right

6 min read 0x.run

I've seen file upload implementations that stored files in the database as BLOBs, routed 500MB video files through Node.js servers, trusted the Content-Type header from the client, and named stored files with the original filename from the user's machine.

Every one of those is a mistake. Let's do it correctly.

The Wrong Approach

// Server-side upload handler — DON'T DO THIS
app.post('/upload', upload.single('file'), async (req, res) => {
    const file = req.file;

    // Wrong: trusting client-provided filename
    const filename = file.originalname;

    // Wrong: trusting client-provided content type
    const type = file.mimetype;

    // Wrong: storing on local disk (no redundancy, fills up, not scalable)
    // Wrong: using user-provided filename (path traversal attack)
    fs.writeFileSync(`/uploads/${filename}`, file.buffer);

    // Wrong: storing in database (bloats DB, slow queries)
    await db.query('INSERT INTO files (data) VALUES ($1)', [file.buffer]);

    res.json({ url: `/uploads/${filename}` });
});

Problems: server handles all file I/O (slow, ties up connections), filename is attacker-controlled, type is attacker-controlled, disk fills up, doesn't scale horizontally.

The Right Architecture: Presigned URLs

Route files directly from the browser to object storage. Your server only issues upload permissions.

BAD:   Browser → Your Server → S3
GOOD:  Browser → S3 directly (your server only issues the presigned URL)

// Server: issue an upload permission
app.post('/upload/presign', requireAuth, async (req, res) => {
    const { filename, contentType, contentLength } = req.body;

    // Validate before issuing permission
    const allowed = validateUploadRequest(filename, contentType, contentLength);
    if (!allowed.ok) {
        return res.status(400).json({ error: allowed.error });
    }

    // Generate a random key — never use the user's filename
    const key = `uploads/${req.user.id}/${crypto.randomUUID()}/${sanitizeFilename(filename)}`;

    const { url, fields } = await s3.createPresignedPost({
        Bucket: process.env.S3_BUCKET,
        Fields: {
            key,
            'Content-Type': contentType,
        },
        Conditions: [
            ['content-length-range', 0, 10 * 1024 * 1024], // 10MB max
            ['eq', '$Content-Type', contentType],
        ],
        Expires: 300, // URL valid for 5 minutes
    });

    // Record pending upload
    const uploadId = await db.createPendingUpload({
        userId: req.user.id,
        key,
        filename: sanitizeFilename(filename),
        status: 'pending',
    });

    res.json({ uploadId, url, fields });
});

// Browser: upload directly to S3
async function uploadFile(file) {
    // Step 1: Get presigned URL from your server
    const { uploadId, url, fields } = await fetch('/upload/presign', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({
            filename: file.name,
            contentType: file.type,
            contentLength: file.size,
        }),
    }).then(r => r.json());

    // Step 2: Upload directly to S3 (no traffic through your server)
    const formData = new FormData();
    Object.entries(fields).forEach(([k, v]) => formData.append(k, v));
    formData.append('file', file); // Must be last

    const uploadResponse = await fetch(url, {
        method: 'POST',
        body: formData,
    });

    if (!uploadResponse.ok) {
        throw new Error('Upload failed');
    }

    // Step 3: Confirm with your server
    await fetch(`/upload/${uploadId}/confirm`, { method: 'POST' });
}

Validate File Type by Magic Bytes

File extensions and Content-Type headers are both user-controlled. Validate by reading the actual file contents.

// Magic byte signatures for common types
const SIGNATURES = {
    'image/jpeg': [[0xFF, 0xD8, 0xFF]],
    'image/png':  [[0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]],
    'image/gif':  [[0x47, 0x49, 0x46, 0x38, 0x37, 0x61],  // GIF87a
                   [0x47, 0x49, 0x46, 0x38, 0x39, 0x61]], // GIF89a
    'image/webp': [[0x52, 0x49, 0x46, 0x46]], // RIFF (need to also check bytes 8-12)
    'application/pdf': [[0x25, 0x50, 0x44, 0x46]],
};

async function detectFileType(buffer) {
    const bytes = new Uint8Array(buffer.slice(0, 16));

    for (const [mimeType, signatures] of Object.entries(SIGNATURES)) {
        for (const sig of signatures) {
            if (sig.every((byte, i) => bytes[i] === byte)) {
                return mimeType;
            }
        }
    }
    return null;
}

// Validate after S3 upload via S3 event → Lambda → your API
async function validateUpload(s3Key) {
    const object = await s3.getObject({
        Bucket: process.env.S3_BUCKET,
        Key: s3Key,
        Range: 'bytes=0-15',  // Only read the header bytes
    });

    const buffer = await object.Body.transformToByteArray();
    const detectedType = await detectFileType(buffer.buffer);

    if (!ALLOWED_TYPES.includes(detectedType)) {
        // Delete the file and reject
        await s3.deleteObject({ Bucket: process.env.S3_BUCKET, Key: s3Key });
        throw new Error(`Rejected: detected type ${detectedType}`);
    }

    return detectedType;
}

Use the file-type npm package if you don't want to maintain magic bytes manually — it covers hundreds of formats.

Enforce Limits at the Right Layer

# nginx.conf — enforce limits before Node sees the request
client_max_body_size 10m;
client_body_timeout 30s;

// Express — secondary check
app.use(express.json({ limit: '1mb' }));

// multer — for multipart uploads going through your server
const upload = multer({
    limits: {
        fileSize: 10 * 1024 * 1024,  // 10MB
        files: 5,                      // Max files per request
        fields: 10,                    // Max non-file fields
    },
});

For presigned uploads: enforce in the presigned URL conditions (as shown above). S3 rejects uploads that exceed the limit before they're stored.

Generate Safe Keys

function sanitizeFilename(original) {
    // Strip path components (defense against ../../../etc/passwd)
    const basename = path.basename(original);

    // Keep only safe characters
    return basename
        .replace(/[^a-zA-Z0-9._-]/g, '_')
        .replace(/_{2,}/g, '_')
        .slice(0, 200);  // Max length
}

function generateStorageKey(userId, originalFilename) {
    const sanitized = sanitizeFilename(originalFilename);
    const uuid = crypto.randomUUID();
    const date = new Date().toISOString().slice(0, 10);  // YYYY-MM-DD

    // Structure: user partitioned, date organized, UUID for uniqueness
    return `uploads/${userId}/${date}/${uuid}/${sanitized}`;
}

Never use the original filename as the storage key. User-provided filenames contain path traversal attempts, null bytes, special characters, and filenames like ../../.env.

Process Uploads Asynchronously

Don't make the user wait for virus scanning, image resizing, or metadata extraction:

// After confirming upload:
app.post('/upload/:id/confirm', requireAuth, async (req, res) => {
    const upload = await db.getPendingUpload(req.params.id, req.user.id);
    if (!upload) return res.status(404).json({ error: 'Not found' });

    // Mark as processing and return immediately
    await db.updateUpload(upload.id, { status: 'processing' });

    // Queue background jobs
    await queue.add('validate-upload', { uploadId: upload.id, key: upload.key });
    await queue.add('generate-thumbnails', { uploadId: upload.id });

    res.json({ status: 'processing', uploadId: upload.id });
});

// Poll for status
app.get('/upload/:id/status', requireAuth, async (req, res) => {
    const upload = await db.getUpload(req.params.id, req.user.id);
    res.json({ status: upload.status, url: upload.publicUrl });
});

Serving Files Securely

Never serve user uploads from the same domain as your app — XSS from uploaded HTML/SVG files would have access to your cookies.

// Serve from a separate domain or use pre-signed read URLs
app.get('/files/:id', requireAuth, async (req, res) => {
    const file = await db.getFile(req.params.id);

    // Check user has permission to this file
    if (!canAccess(req.user, file)) {
        return res.status(403).json({ error: 'Forbidden' });
    }

    // Generate a short-lived presigned GET URL
    const url = await s3.getSignedUrl('getObject', {
        Bucket: process.env.S3_BUCKET,
        Key: file.key,
        Expires: 3600,  // 1 hour
        ResponseContentDisposition: `attachment; filename="${file.filename}"`,
    });

    // Redirect to S3 (client fetches directly)
    res.redirect(url);
});

The Bottom Line

File uploads touch security, performance, and reliability simultaneously. The naive implementation gets all three wrong.

The rules:

  • Presigned URLs: browsers upload directly to S3/R2, not through your server
  • Validate by magic bytes, not extension or Content-Type header
  • Generate storage keys — never use user-provided filenames
  • Enforce size limits at nginx/load balancer, not just application code
  • Serve uploads from a separate domain or with presigned read URLs
  • Process async: validate, resize, scan in background jobs
  • Set expiry on presigned upload URLs (5 minutes is plenty)

Most of the complexity here is one-time setup. Once the pattern is in place, adding new upload types is straightforward.