cd ..

$ cat ~/field-notes/secrets-management-and-rotation.md

Secrets Management Beyond .env Files

6 min read 0x.run

Putting DATABASE_PASSWORD in an environment variable is better than hardcoding it in Git. It does not answer the harder questions:

  • Who is allowed to read it?
  • How did it reach the process?
  • Where is it logged or copied?
  • How quickly can it be replaced?
  • What happens to live traffic during rotation?

Secret storage is only one part of secret management.

What Counts as a Secret

Secrets grant authority:

database passwords
API keys
private signing keys
OAuth client secrets
encryption keys
session signing material
webhook signing secrets
service account credentials

A hostname, feature flag, or public certificate is configuration, not a secret. Classifying everything as secret creates needless operational friction. Classifying a private key as ordinary configuration creates an incident.

The Four Parts of the System

Separate these concerns:

storage       encrypted secret values and versions
identity      proof of which workload is asking
authorization policy deciding what that identity may read
delivery      how the value reaches the application

A secrets manager encrypted with a strong key is still unsafe if every workload can call GetSecret(*).

Prefer narrow paths and roles:

payments production workload:
  can read production/payments/database
  can read production/payments/provider-key
  cannot list all secrets
  cannot read staging or another service's credentials

Audit both successful and denied access without recording the secret value.

Solve the Secret-Zero Problem with Identity

If an application needs a static API key to fetch its real secrets, that bootstrap key becomes "secret zero." It must be stored, delivered, and rotated too.

Cloud instance identity, container workload identity, machine certificates, or an orchestrator service account can authenticate the workload without a long-lived key in the repository or image.

workload proves platform identity
        ↓
platform issues short-lived access token
        ↓
secrets manager authorizes exact secret paths

Keep the bootstrap trust anchored in the platform. Do not bake a universal secrets-manager token into a base image.

Prefer Dynamic Credentials

The best long-lived password is one you do not have.

Some systems can issue credentials on demand:

application authenticates by workload identity
secrets system creates database user with 30-minute lease
application connects
lease expires or is revoked

Short-lived credentials reduce the useful lifetime of a leak and make attribution clearer. They also require applications to refresh credentials and connection pools correctly.

For databases, a pool may keep authenticated connections after the credential expires while new connections fail. Refresh before expiry, update the pool's credential source, and gradually replace old connections. Test the actual driver and proxy behavior.

Environment Variables vs Mounted Files

Environment variables are simple and widely supported. They also tend to be copied into crash reports, debug pages, child processes, deployment manifests, and support bundles.

Mounted secret files can have tighter filesystem permissions and can be updated atomically by an agent or orchestrator. Applications must notice file changes and reload them safely.

Neither delivery method is automatically secure:

environment variable + narrow identity + redacted diagnostics
    can be reasonable

mounted file + world-readable permissions + copied into image layer
    is not secure

Choose based on runtime support and threat model. Minimize copies either way.

Cache Without Making Secrets Permanent

Fetching a secret for every request adds latency and makes the secrets service part of every request path. Cache values in memory with a bounded refresh interval.

class SecretCache {
    constructor(fetchSecret, ttlMs = 5 * 60_000) {
        this.fetchSecret = fetchSecret;
        this.ttlMs = ttlMs;
        this.value = null;
        this.expiresAt = 0;
        this.inFlight = null;
    }

    async get() {
        if (this.value && Date.now() < this.expiresAt) return this.value;

        // Coalesce concurrent refreshes.
        if (!this.inFlight) {
            this.inFlight = this.fetchSecret()
                .then(value => {
                    this.value = value;
                    this.expiresAt = Date.now() + this.ttlMs;
                    return value;
                })
                .finally(() => {
                    this.inFlight = null;
                });
        }

        return this.inFlight;
    }
}

Add jitter so every instance does not refresh at once. Decide whether a fetch failure may use the last known value briefly or must fail closed; the right answer depends on the credential's revocation semantics.

Never write the cache to application logs or a generic local cache shared with normal data.

Zero-Downtime Rotation Needs Overlap

Replacing a secret instantaneously fails because consumers do not update at the same instant.

A safe static-key rotation has phases:

1. create new credential; old still works
2. distribute or make new version available
3. update and verify every consumer
4. observe that old credential is no longer used
5. revoke old credential
6. remove old version after audit window

For signing or webhook verification, accept both old and new keys during the overlap but sign new data only with the new key.

function verifySignature(payload, signature, keys) {
    return keys.some(key => constantTimeVerify(payload, signature, key));
}

Label keys with versions or IDs so verifiers can choose the right one without trying an unbounded key list.

Rotation without removal is accumulation. The process is complete only when the old credential is revoked.

Prevent Accidental Exfiltration

Most secret leaks are not cryptographic failures. They are ordinary data handling failures.

// Bad
logger.info({ config: process.env }, 'starting application');
logger.error({ requestHeaders: req.headers }, 'request failed');

// Better: explicit allowlist
logger.info({
    port: config.port,
    environment: config.environment,
    databaseHost: config.databaseHost,
}, 'starting application');

Use allowlists rather than trying to redact an ever-growing list of secret field names. Scrub authorization headers, cookies, URL query strings, database URLs, and command-line arguments from diagnostics.

Secret scanners in pre-commit hooks and CI catch common patterns, but a clean scan is not proof. A random token may have no recognizable prefix.

When a Secret Leaks

Do not merely delete it from the latest commit.

1. revoke or rotate immediately
2. identify scope and permissions
3. inspect access and use logs
4. remove it from reachable history and artifacts where practical
5. invalidate derived credentials or sessions if necessary
6. fix the delivery or logging path that leaked it

Assume a committed secret was copied, even in a private repository. Git history rewriting reduces future exposure; revocation ends current authority.

The Bottom Line

A secret is managed only when its identity, access, delivery, lifetime, and retirement are controlled.

The rules:

  • Use workload identity instead of a static bootstrap key
  • Grant access to exact secret paths with least privilege
  • Prefer short-lived dynamic credentials where supported
  • Cache in memory briefly and refresh before expiry
  • Rotate with an overlap period, then revoke the old version
  • Keep secrets out of logs, URLs, images, and command arguments
  • Audit access without logging values
  • Treat revocation—not Git cleanup—as the first incident action

Moving secrets out of source code is the beginning. Making them short-lived and replaceable is what turns a credential leak from a permanent problem into a bounded one.