cd ..

$ cat ~/field-notes/how-containers-work-namespaces-cgroups.md

How Containers Actually Work: Namespaces, cgroups, and Layers

6 min read 0x.run

There is no container object inside the Linux kernel. The kernel sees processes, namespaces, cgroups, mounts, capabilities, and system calls.

A container runtime combines those features into something that looks like a small machine. Once you understand the pieces, container behavior stops feeling magical.

A Container Is an Isolated Process

Compare the models:

Virtual machine:
  hardware → host kernel → hypervisor → guest kernel → process

Container:
  hardware → host kernel → isolated process

Containers share the host kernel. That makes startup fast and memory overhead low, but it also means a container cannot run a kernel for another operating system. A Linux container needs a Linux kernel even if the image contains Ubuntu, Alpine, or no traditional distribution at all.

The files in an Ubuntu image are user-space libraries and tools. They are not an Ubuntu kernel.

Namespaces Control What a Process Sees

Linux namespaces give a process a restricted view of global resources.

PID namespace      process IDs and process tree
mount namespace    filesystem mount table
network namespace  interfaces, routes, ports, firewall rules
UTS namespace      hostname and domain name
IPC namespace      shared memory and message queues
user namespace     user/group ID mappings
cgroup namespace   cgroup path view
time namespace     selected system clocks

Inside a PID namespace, an application can be PID 1. On the host, that same process has another PID.

inside container: PID 1
host view:        PID 28741
same process, two namespace-relative identifiers

A network namespace explains why two containers can both listen on port 8080. Each has its own loopback interface and port table. Publishing a port creates a path from the host network into one container namespace.

Build a Namespace by Hand

On a Linux system with suitable permissions, unshare starts a process in new namespaces:

sudo unshare \
  --fork \
  --pid \
  --mount-proc \
  --uts \
  --mount \
  /bin/bash

# Inside the new PID namespace:
echo $$
# 1

hostname demo-container
ps aux

The shell sees itself as PID 1 and can change its namespaced hostname without changing the host hostname. This is not a complete or secure container—there is no isolated root filesystem, network setup, capability reduction, or resource limit—but it reveals the basic primitive.

cgroups Control What a Process Can Use

Namespaces answer "what can I see?" Control groups answer "how much can I use?"

cgroups group processes and account for or limit resources such as:

CPU time and weight
memory and swap
number of processes
I/O bandwidth and priority

On a cgroup v2 system, a memory limit is represented by files such as:

cat /sys/fs/cgroup/memory.max
cat /sys/fs/cgroup/memory.current
cat /sys/fs/cgroup/memory.events

The exact visible path depends on the runtime and cgroup namespace.

A container with a 512 MB memory limit does not see a virtual 512 MB RAM stick. Its processes allocate normal host memory, and the kernel accounts that usage to the cgroup. When the group exceeds its limit and reclaim cannot recover enough, the kernel may invoke an out-of-memory kill within the group.

CPU Limits Are Time, Not Smaller CPUs

A CPU quota gives a cgroup a budget over a period. Conceptually:

50 ms of CPU time per 100 ms period = 0.5 CPU
200 ms per 100 ms period           = 2 CPUs

If runnable processes consume the budget early, the cgroup is throttled until the next period. This can create latency spikes even when host-wide CPU graphs do not show 100% usage.

Monitor cgroup throttling, not just overall CPU percentage. An application capped at one CPU can be saturated on a 32-core host that looks mostly idle.

CPU requests or weights and hard CPU limits solve different problems. A weight controls relative competition under contention; a quota sets a ceiling.

Images Are Layered Filesystems

An image is a stack of read-only layers plus metadata:

read-only: application files       ← COPY . /app
read-only: installed dependencies  ← RUN npm ci
read-only: runtime files           ← base image
---------------------------------
writable: container changes

Overlay filesystems present those layers as one directory tree. If a container modifies a file from a read-only layer, copy-on-write places a changed copy in the writable layer.

This explains several behaviors:

  • Multiple containers can share image layers on disk
  • Deleting a large file in a later image layer does not remove its bytes from earlier layers
  • Writing heavily into the container layer can be slower than using a volume
  • Removing the container removes its writable layer

Image layers are not a backup system. Persistent databases and uploads belong in managed storage or explicit volumes with their own backup policy.

A Dockerfile Layer Trap

# Bad: secret exists forever in an earlier image layer.
COPY .npmrc /root/.npmrc
RUN npm ci
RUN rm /root/.npmrc

The delete creates a later layer that hides the file. It does not erase the file from the layer created by COPY.

Use build-time secret mounts supported by the builder, and make sure the secret is never copied into an image layer:

# syntax=docker/dockerfile:1
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc \
    npm ci

Also keep build context small with .dockerignore. Files sent to the builder are not automatically image layers, but they still cross a trust boundary and can be copied accidentally.

Root in a Container Is Still Powerful

UID 0 inside a container is not automatically host root, but it has special power within its namespaces and increases the impact of a kernel or runtime escape.

Reduce that power in layers:

RUN addgroup --system app && adduser --system --ingroup app app
USER app

At runtime:

docker run \
  --read-only \
  --cap-drop ALL \
  --security-opt no-new-privileges \
  --tmpfs /tmp:rw,noexec,nosuid,size=64m \
  my-app

Add back only a capability the application demonstrably needs. Linux capabilities split traditional root power into units such as changing ownership, binding privileged ports, or administering networks.

Seccomp adds another boundary by filtering system calls. Mandatory access control systems can constrain file and process access further. Rootless runtimes and user namespaces map container IDs to unprivileged host IDs.

No single control is the sandbox. The defense is cumulative.

The Runtime's Job

At a high level, a container runtime:

1. pulls and verifies image content
2. assembles the root filesystem layers
3. creates namespaces
4. creates and configures a cgroup
5. applies capabilities, seccomp, and other security policy
6. sets mounts, environment, user, and working directory
7. starts the requested process
8. monitors and reports its exit

Higher-level systems add image distribution, networking, volume management, restarts, scheduling, and health checks. The final workload is still a host process governed by kernel primitives.

Debug at the Right Layer

# Container configuration and state
docker inspect my-container

# Processes as Docker sees them
docker top my-container

# Live resource counters
docker stats my-container

# Namespace links for a host PID
ls -l /proc/28741/ns/

# cgroup membership for a host PID
cat /proc/28741/cgroup

If a process is killed at its memory limit, increasing an application heap setting beyond the cgroup limit cannot help. If it cannot bind a port, inspect user and capabilities. If a file disappears after recreation, inspect the mount and writable layer.

The Bottom Line

Containers are composed from ordinary Linux isolation and resource-control features.

The rules:

  • Remember that containers share the host kernel
  • Use namespaces to reason about visibility and identity
  • Use cgroup metrics to understand CPU throttling and memory kills
  • Treat the writable image layer as disposable
  • Keep secrets out of image layers and build context
  • Run as a non-root user and drop unused capabilities
  • Prefer read-only filesystems and explicit writable mounts
  • Debug the host process, namespaces, cgroup, and mounts separately

"A container is a process" is simplified, but it is the right place to start. The rest is the set of kernel boundaries built around that process.