inside containerd: how shim and runc actually work

k8scontainerinfrastructure

kubelet calls containerd, and a container appears. But between that call and the running container, three processes do a relay. One of them is created to die.

Think of it as looking after a newborn. Each role maps to a real process:

Role in the pictureProcessWhat it does
The childThe program in the container, e.g. nginxNeeds someone looking after it all the time
The doctor who delivers itruncDelivers the child, then leaves
The parentscontainerdHave jobs and are sometimes away (upgrades, restarts)
The nannycontainerd-shim-runc-v2Stays with the child and phones the parents when something happens
The agency that sends the nannycontainerd-shim-runc-v2 startBrings the nanny, leaves her phone number, and goes
Social servicessystemd (PID 1)Takes in every child with no parent

A PID is the number Linux gives each process; PID 1 is the first process after boot, usually systemd.

The whole relay:

kubectl set image deployment/api api=api:v2
  │
  ① kubectl → API Server → etcd (write new image)
  ② Deployment Controller → creates new ReplicaSet
  ③ ReplicaSet Controller → creates Pod objects
  ④ Scheduler → assigns Pod to a Node
  ⑤ kubelet picks up the Pod
  │
  ▼
containerd
  │ fork + exec
  ▼
containerd-shim-runc-v2 start   (the agency, short-lived)
  │ re-execs itself, then exits (double fork)
  ▼
containerd-shim-runc-v2 (the nanny) ← reparented to PID 1
  │ fork + exec (runc runs as the shim's child; the shim stays)
  ▼
runc
  │ set up namespaces, cgroups, pivot_root
  │ start container process
  │ exit (job done)
  ▼
container process (running)
  parent = containerd-shim-runc-v2
  grandparent = systemd (PID 1)
Steady state (the start launcher and runc both exited):

systemd (PID 1)
  ├── containerd                         ← the parents, talk over a ttrpc socket
  └── containerd-shim-runc-v2           ← the nanny, asleep until SIGCHLD
          └── nginx (container PID 1)    ← the child

containerd and containerd-shim-runc-v2 are siblings under systemd, not parent and child. They talk over a ttrpc socket, the way the nanny phones the parents, not through the parent-child relationship between processes. nginx shows PID 1 inside the container's PID namespace. From the host, it has a normal PID like 50321.

This is containerd's internal machinery. K8s doesn't know any of this exists. K8s needed Docker. Then it didn't. covers how K8s got to containerd in the first place.


runc: delivers, then leaves

runc is a CLI tool that builds containers. It sets up Linux namespaces (what the container can see) and cgroups (how much CPU and memory it can use), switches the root filesystem with pivot_root (it falls back to chroot only when run with --no-pivot), starts the container's program, and then exits.

The container keeps running even though runc has already exited. Every process has a parent, the process that created it. So which process is the container's parent now?


Why each layer exists

The container needs a long-lived parent that does three things:

  • Holds its stdin, stdout and stderr and passes the output to containerd, whose CRI logger writes it to a log file on the node; kubectl logs reads that file
  • Calls waitpid() when the container dies and passes the exit code up, so kubectl get pod can show Completed or Error
  • Answers containerd's queries, such as kubectl exec and status checks

runc can't do this. It already exited.

Could containerd be the parent? It could, but containerd gets restarted for version upgrades, security patches and config changes. If containerd were the container's parent, then when containerd exits the container is orphaned and adopted by PID 1, and the restarted containerd is no longer its parent and can't collect its exit code. So the design came out one problem at a time (this is the order problems were discovered, not the order processes run):

  1. Container needs a long-lived parent → runc can't stay
  2. Make containerd the parent? → containerd restarts too often
  3. Create a dedicated process to look after the container → containerd-shim-runc-v2, the nanny
  4. The nanny can't sit under containerd → needs double fork
  5. Double fork needs a short-lived middle process that creates the nanny and exits → the shim binary runs a first time with the start argument just to do that, the agency

The orphan problem

Every Linux process has a parent. When a child dies, the parent must call wait() to clean it up. Without wait(), the dead child becomes a zombie: it no longer runs, and what's left is its entry in the process table, which takes a little memory and holds on to a PID. Pile up zombies and PIDs are what run out first.

A living orphan is worse. When a parent dies, the kernel reparents its children to systemd (PID 1). systemd keeps them alive but doesn't know if they should be alive. A runaway orphan process eats CPU and RAM until someone kills it or it exits on its own.

containerd-shim-runc-v2 is the container's parent. But who parents containerd-shim-runc-v2? And how do you detach it from containerd?


Double fork: the agency brings the nanny and leaves

The two actors in a double fork are one binary, containerd-shim-runc-v2, run twice, not the two binaries containerd-shim and containerd-shim-runc-v2:

containerd-shim-runc-v2 startcontainerd-shim-runc-v2 (no args)
LifespanLess than a secondLives as long as the container
PurposeRe-exec the real shim, report its socket address to containerd, then exitLook after the container
Role in the pictureThe agencyThe nanny

containerd-shim without the suffix is a different thing, the old v1 shim binary, covered at the end.

Double fork is a classic Unix trick: two forks in a row, where the middle process exits. Fork makes a process copy itself into a child; exec makes a process switch to running another program.

  1. containerd forks and execs → containerd-shim-runc-v2 start (child)
  2. the start process re-execs the same binary → containerd-shim-runc-v2 daemon (grandchild), in a new process group
  3. the start process reports the daemon's socket address to containerd, then exits
  4. the daemon's parent is gone, so it is an orphan
  5. the kernel reparents the daemon to PID 1 (systemd)

Step 5 happens because the launcher exits in step 3, not because of the new process group in step 2. A process is adopted by PID 1 only when its own parent dies. The launcher's whole job is to die at once, so the daemon it created is adopted by PID 1 right away, while containerd is still alive. The new process group is a separate thing, covered below.

A common belief is that a child dies when its parent dies. Linux doesn't work that way.

Parent dies → child gets adopted by PID 1 → child keeps running.

The belief comes from shells. As the bash manual describes, when a terminal is closed, bash receives SIGHUP, and before exiting it resends SIGHUP to its jobs; the default response to SIGHUP is to quit. Those children die because bash sends them a signal, not because the parent died. That's shell behavior, not the kernel's.

The launcher exits without sending any signal, and it's not a shell, so the daemon just becomes an orphan and keeps running.

Before (right after fork):

  containerd (PID 500)
    └── containerd-shim-runc-v2 start (PID 501)
          └── containerd-shim-runc-v2 (PID 502)
After (the start process exits):

  containerd (PID 500)              ← no longer parent of anything
  systemd (PID 1)
    └── containerd-shim-runc-v2 (PID 502)   ← adopted by PID 1
          └── container process (PID 503)

From here on, containerd and containerd-shim-runc-v2 are not parent and child. A socket can disconnect and reconnect; the parent-child relationship can't be reconnected that way.

Besides leaving the parent-child relationship, the shim uses two more ways to stay clear of containerd:

ScopeToolProtects against
parent-child relationshipreparent to PID 1losing the parent, and with it the waitpid relationship
process groupsetpgidCtrl+C or kill -<pgid>, signals sent to a whole group
cgroupKillMode=processsystemctl stop killing every process in the cgroup
Deeper: why not just fork once?

Fork once and the shim is containerd's direct child, which is how v1 shims work (see the last section). Nothing crashes, since systemd still reaps orphans, but containerd takes on work it shouldn't: every shim that exits sends it a SIGCHLD, and its child list fills up with shims. After a restart, the new containerd can't waitpid() shims that are no longer its children; Linux has no call to reassign a parent.

You might think fork-once ends up in the same place, since the shim is adopted by PID 1 whenever containerd eventually exits. It does in the end, but until then containerd has to treat every shim it created as a child, so all the work above lands on containerd. Double fork has each shim adopted by PID 1 the moment it starts, so containerd never has to look after them.

Inferred from the double fork pattern, the iximiuz shim walkthrough, and the Container42 explainer; no containerd doc spells this out.

Deeper: the process-group and cgroup rows

The table's second row is about signals. The daemon starts in its own process group via setpgid (containerd's manager_linux.go sets Setpgid: true). Group signals are delivered by process group, so if the shim shared containerd's group, a kill -<pgid> (or a Ctrl+C, if containerd ran in a foreground terminal) would reach the shim too.

The third row, cgroup, is separate again. setpgid changes only the process group, not the cgroup, and by default systemctl stop kills every process in the service's cgroup, which includes the shims. That's why containerd.service ships with KillMode=process: stopping containerd then kills only containerd, not the shims sharing its cgroup.


What containerd-shim-runc-v2 actually does all day

After runc exits and the container is running, containerd-shim-runc-v2 stays and does three things:

JobChannelWhy
Hold stdin/stdout/stderrfile descriptorsOutput goes through containerd into the log file kubectl logs reads; kubectl attach also connects through it
Call wait() when the container diesparent-child relationshipCollects the exit code so kubectl get pod can show Completed or Error
Report to containerd and respond to commandsttrpc socketcontainerd asks "is it still running?" or "kill it"

Only the parent can collect a child's exit code, so the shim's waitpid() relies on the parent-child relationship, and that code flows up through containerd to kubelet as the Pod status you see in kubectl get pod. Reporting takes the other channel, a ttrpc socket that can break and reconnect.

If containerd crashes, containerd-shim-runc-v2 still collects the exit code, because the parent-child relationship is intact. It just can't report yet, like a nanny phoning parents who don't pick up. When the new containerd reconnects the socket, the shim delivers the result. Collecting the exit code and reporting it don't depend on each other.

Most of the time containerd-shim-runc-v2 is idle, but idle here means waiting, not polling. The shim adds SIGCHLD to the signals it handles. When any of its children dies, the kernel sends the shim one SIGCHLD, and the shim calls reaper.Reap(), which loops over wait4(-1, ..., WNOHANG) to collect every child that has exited, whichever container it belongs to, then hands each exit status to whoever subscribed. The code is in pkg/shim/shim_unix.go and pkg/sys/reaper/reaper_unix.go. A Pod with several containers still means one signal handler, not one waiter per container, so the shim stays responsive on the socket the whole time.

Deeper: why the container reparents to the shim, not PID 1

There's one gap left. runc started the container, then exited. By the orphan rule above, the container should now be adopted by PID 1 like any other orphan. If it were, the shim could no longer waitpid() it, and the exit code would be lost.

The shim prevents that by marking itself a subreaper when it starts:

prctl(PR_SET_CHILD_SUBREAPER, 1)

That changes the rule for all its descendants. When runc exits, the container isn't handed all the way up to PID 1; it goes to the nearest ancestor that set the subreaper flag, which is the shim. In the picture, the nanny has registered as guardian, so when the doctor leaves, the child stays with her instead of going to social services. The container stays under the shim, and waitpid() keeps working.

The shim has no subreaper above it, so it goes all the way up to PID 1, away from containerd. The container has the shim above it as a subreaper, so it stays put.

Deeper: kernel vs systemd, who notifies whom

The kernel and systemd are not the same thing. The kernel is the OS core running in kernel space. systemd is a regular process (PID 1) running in user space.

The notification always comes from the kernel. systemd's only role is being the last-resort parent for orphans:

EventWho handles it
container dieskernel wakes containerd-shim-runc-v2
containerd-shim-runc-v2 dieskernel wakes systemd
systemd dieskernel panic, machine is dead

systemd is a passive adopter. containerd-shim-runc-v2 is an active guardian. systemd doesn't know or care what containerd-shim-runc-v2 is; it just calls wait() when an adopted child dies. containerd-shim-runc-v2 knows exactly what it's watching and reports the exit code back to containerd.


v1 → v2: double fork added, fewer processes

Both v1 and v2 shims hold stdio, collect exit codes with wait, and mark themselves as subreapers. v1 has no double fork, though. In v1.2.0's client.go and shim_linux.go, containerd starts containerd-shim directly as its child and Waits on it, and the shim is adopted by PID 1 only when containerd exits. So on a node still running v1, the shim sits under containerd, not under PID 1. v2 added the start step, and also changed how many shim processes you run:

v1: one containerd-shim per container

Under CRI every Pod also has a sandbox (pause) container, and openEuler's comparison of shim v1 and v2 notes that under v1 it got its own shim too.

  • Pod with 3 containers → 4 containerd-shim processes (3 + the pause container)
  • 30 Pods x 2 containers average → 90 containerd-shim processes on one Node

v2: one containerd-shim-runc-v2 per Pod

  • Pod with 3 containers → 1 containerd-shim-runc-v2 process managing all three
  • 30 Pods → 30 containerd-shim-runc-v2 processes (not 90)

Both versions talk to containerd over ttrpc, containerd's lightweight take on gRPC. Per the ttrpc README, it drops the net/http, HTTP/2 and grpc packages for a simple framing protocol, so binaries are smaller and use less memory. v2's win is the process count, not the transport. gRPC for shims only arrived as an option in containerd 1.7, and containerd's runtime v2 docs still call it experimental and recommend ttrpc.

The naming convention also made runtimes pluggable. containerd turns a runtime_type into a binary name:

  • io.containerd.runc.v2 → containerd-shim-runc-v2 → uses runc (default)
  • io.containerd.runsc.v1 → containerd-shim-runsc-v1 → uses gVisor
  • io.containerd.kata.v2 → containerd-shim-kata-v2 → uses Kata

Adding a runtime takes three steps: put its shim binary and runtime binary on the Node, register a handler with that runtime_type in containerd's config, and create a RuntimeClass that names the handler. A Pod that sets runtimeClassName then gets that shim. When the kernel has a bug, what keeps containers apart? walks through the gVisor version.


References