inside containerd: how shim and runc actually work
kubelet calls containerd, and a container appears. But between that call and the running container, three processes do a relay. One of them is created to die.
Think of it as looking after a newborn. Each role maps to a real process:
| Role in the picture | Process | What it does |
|---|---|---|
| The child | The program in the container, e.g. nginx | Needs someone looking after it all the time |
| The doctor who delivers it | runc | Delivers the child, then leaves |
| The parents | containerd | Have jobs and are sometimes away (upgrades, restarts) |
| The nanny | containerd-shim-runc-v2 | Stays with the child and phones the parents when something happens |
| The agency that sends the nanny | containerd-shim-runc-v2 start | Brings the nanny, leaves her phone number, and goes |
| Social services | systemd (PID 1) | Takes in every child with no parent |
A PID is the number Linux gives each process; PID 1 is the first process after boot, usually systemd.
The whole relay:
kubectl set image deployment/api api=api:v2
│
① kubectl → API Server → etcd (write new image)
② Deployment Controller → creates new ReplicaSet
③ ReplicaSet Controller → creates Pod objects
④ Scheduler → assigns Pod to a Node
⑤ kubelet picks up the Pod
│
▼
containerd
│ fork + exec
▼
containerd-shim-runc-v2 start (the agency, short-lived)
│ re-execs itself, then exits (double fork)
▼
containerd-shim-runc-v2 (the nanny) ← reparented to PID 1
│ fork + exec (runc runs as the shim's child; the shim stays)
▼
runc
│ set up namespaces, cgroups, pivot_root
│ start container process
│ exit (job done)
▼
container process (running)
parent = containerd-shim-runc-v2
grandparent = systemd (PID 1)
Steady state (the start launcher and runc both exited):
systemd (PID 1)
├── containerd ← the parents, talk over a ttrpc socket
└── containerd-shim-runc-v2 ← the nanny, asleep until SIGCHLD
└── nginx (container PID 1) ← the child
containerd and containerd-shim-runc-v2 are siblings under systemd, not parent and child. They talk over a ttrpc socket, the way the nanny phones the parents, not through the parent-child relationship between processes. nginx shows PID 1 inside the container's PID namespace. From the host, it has a normal PID like 50321.
This is
containerd's internal machinery. K8s doesn't know any of this exists. K8s needed Docker. Then it didn't. covers how K8s got tocontainerdin the first place.
runc: delivers, then leaves
runc is a CLI tool that builds containers. It sets up Linux namespaces (what the container can see) and cgroups (how much CPU and memory it can use), switches the root filesystem with pivot_root (it falls back to chroot only when run with --no-pivot), starts the container's program, and then exits.
The container keeps running even though runc has already exited. Every process has a parent, the process that created it. So which process is the container's parent now?
Why each layer exists
The container needs a long-lived parent that does three things:
- Holds its stdin, stdout and stderr and passes the output to
containerd, whose CRI logger writes it to a log file on the node;kubectl logsreads that file - Calls
waitpid()when the container dies and passes the exit code up, sokubectl get podcan showCompletedorError - Answers
containerd's queries, such askubectl execand status checks
runc can't do this. It already exited.
Could containerd be the parent? It could, but containerd gets restarted for version upgrades, security patches and config changes. If containerd were the container's parent, then when containerd exits the container is orphaned and adopted by PID 1, and the restarted containerd is no longer its parent and can't collect its exit code. So the design came out one problem at a time (this is the order problems were discovered, not the order processes run):
- Container needs a long-lived parent →
runccan't stay - Make
containerdthe parent? →containerdrestarts too often - Create a dedicated process to look after the container →
containerd-shim-runc-v2, the nanny - The nanny can't sit under
containerd→ needs double fork - Double fork needs a short-lived middle process that creates the nanny and exits → the shim binary runs a first time with the
startargument just to do that, the agency
The orphan problem
Every Linux process has a parent. When a child dies, the parent must call wait() to clean it up. Without wait(), the dead child becomes a zombie: it no longer runs, and what's left is its entry in the process table, which takes a little memory and holds on to a PID. Pile up zombies and PIDs are what run out first.
A living orphan is worse. When a parent dies, the kernel reparents its children to systemd (PID 1). systemd keeps them alive but doesn't know if they should be alive. A runaway orphan process eats CPU and RAM until someone kills it or it exits on its own.
containerd-shim-runc-v2 is the container's parent. But who parents containerd-shim-runc-v2? And how do you detach it from containerd?
Double fork: the agency brings the nanny and leaves
The two actors in a double fork are one binary, containerd-shim-runc-v2, run twice, not the two binaries containerd-shim and containerd-shim-runc-v2:
containerd-shim-runc-v2 start | containerd-shim-runc-v2 (no args) | |
|---|---|---|
| Lifespan | Less than a second | Lives as long as the container |
| Purpose | Re-exec the real shim, report its socket address to containerd, then exit | Look after the container |
| Role in the picture | The agency | The nanny |
containerd-shim without the suffix is a different thing, the old v1 shim binary, covered at the end.
Double fork is a classic Unix trick: two forks in a row, where the middle process exits. Fork makes a process copy itself into a child; exec makes a process switch to running another program.
containerdforks and execs →containerd-shim-runc-v2 start(child)- the
startprocess re-execs the same binary →containerd-shim-runc-v2daemon (grandchild), in a new process group - the
startprocess reports the daemon's socket address tocontainerd, then exits - the daemon's parent is gone, so it is an orphan
- the kernel reparents the daemon to PID 1 (
systemd)
Step 5 happens because the launcher exits in step 3, not because of the new process group in step 2. A process is adopted by PID 1 only when its own parent dies. The launcher's whole job is to die at once, so the daemon it created is adopted by PID 1 right away, while containerd is still alive. The new process group is a separate thing, covered below.
A common belief is that a child dies when its parent dies. Linux doesn't work that way.
Parent dies → child gets adopted by PID 1 → child keeps running.
The belief comes from shells. As the bash manual describes, when a terminal is closed, bash receives SIGHUP, and before exiting it resends SIGHUP to its jobs; the default response to SIGHUP is to quit. Those children die because bash sends them a signal, not because the parent died. That's shell behavior, not the kernel's.
The launcher exits without sending any signal, and it's not a shell, so the daemon just becomes an orphan and keeps running.
Before (right after fork):
containerd (PID 500)
└── containerd-shim-runc-v2 start (PID 501)
└── containerd-shim-runc-v2 (PID 502)
After (the start process exits):
containerd (PID 500) ← no longer parent of anything
systemd (PID 1)
└── containerd-shim-runc-v2 (PID 502) ← adopted by PID 1
└── container process (PID 503)
From here on, containerd and containerd-shim-runc-v2 are not parent and child. A socket can disconnect and reconnect; the parent-child relationship can't be reconnected that way.
Besides leaving the parent-child relationship, the shim uses two more ways to stay clear of containerd:
| Scope | Tool | Protects against |
|---|---|---|
| parent-child relationship | reparent to PID 1 | losing the parent, and with it the waitpid relationship |
| process group | setpgid | Ctrl+C or kill -<pgid>, signals sent to a whole group |
| cgroup | KillMode=process | systemctl stop killing every process in the cgroup |
Deeper: why not just fork once?
Fork once and the shim is containerd's direct child, which is how v1 shims work (see the last section). Nothing crashes, since systemd still reaps orphans, but containerd takes on work it shouldn't: every shim that exits sends it a SIGCHLD, and its child list fills up with shims. After a restart, the new containerd can't waitpid() shims that are no longer its children; Linux has no call to reassign a parent.
You might think fork-once ends up in the same place, since the shim is adopted by PID 1 whenever containerd eventually exits. It does in the end, but until then containerd has to treat every shim it created as a child, so all the work above lands on containerd. Double fork has each shim adopted by PID 1 the moment it starts, so containerd never has to look after them.
Inferred from the double fork pattern, the iximiuz shim walkthrough, and the Container42 explainer; no containerd doc spells this out.
Deeper: the process-group and cgroup rows
The table's second row is about signals. The daemon starts in its own process group via setpgid (containerd's manager_linux.go sets Setpgid: true). Group signals are delivered by process group, so if the shim shared containerd's group, a kill -<pgid> (or a Ctrl+C, if containerd ran in a foreground terminal) would reach the shim too.
The third row, cgroup, is separate again. setpgid changes only the process group, not the cgroup, and by default systemctl stop kills every process in the service's cgroup, which includes the shims. That's why containerd.service ships with KillMode=process: stopping containerd then kills only containerd, not the shims sharing its cgroup.
What containerd-shim-runc-v2 actually does all day
After runc exits and the container is running, containerd-shim-runc-v2 stays and does three things:
| Job | Channel | Why |
|---|---|---|
| Hold stdin/stdout/stderr | file descriptors | Output goes through containerd into the log file kubectl logs reads; kubectl attach also connects through it |
Call wait() when the container dies | parent-child relationship | Collects the exit code so kubectl get pod can show Completed or Error |
Report to containerd and respond to commands | ttrpc socket | containerd asks "is it still running?" or "kill it" |
Only the parent can collect a child's exit code, so the shim's waitpid() relies on the parent-child relationship, and that code flows up through containerd to kubelet as the Pod status you see in kubectl get pod. Reporting takes the other channel, a ttrpc socket that can break and reconnect.
If containerd crashes, containerd-shim-runc-v2 still collects the exit code, because the parent-child relationship is intact. It just can't report yet, like a nanny phoning parents who don't pick up. When the new containerd reconnects the socket, the shim delivers the result. Collecting the exit code and reporting it don't depend on each other.
Most of the time containerd-shim-runc-v2 is idle, but idle here means waiting, not polling. The shim adds SIGCHLD to the signals it handles. When any of its children dies, the kernel sends the shim one SIGCHLD, and the shim calls reaper.Reap(), which loops over wait4(-1, ..., WNOHANG) to collect every child that has exited, whichever container it belongs to, then hands each exit status to whoever subscribed. The code is in pkg/shim/shim_unix.go and pkg/sys/reaper/reaper_unix.go. A Pod with several containers still means one signal handler, not one waiter per container, so the shim stays responsive on the socket the whole time.
Deeper: why the container reparents to the shim, not PID 1
There's one gap left. runc started the container, then exited. By the orphan rule above, the container should now be adopted by PID 1 like any other orphan. If it were, the shim could no longer waitpid() it, and the exit code would be lost.
The shim prevents that by marking itself a subreaper when it starts:
prctl(PR_SET_CHILD_SUBREAPER, 1)
That changes the rule for all its descendants. When runc exits, the container isn't handed all the way up to PID 1; it goes to the nearest ancestor that set the subreaper flag, which is the shim. In the picture, the nanny has registered as guardian, so when the doctor leaves, the child stays with her instead of going to social services. The container stays under the shim, and waitpid() keeps working.
The shim has no subreaper above it, so it goes all the way up to PID 1, away from containerd. The container has the shim above it as a subreaper, so it stays put.
Deeper: kernel vs systemd, who notifies whom
The kernel and systemd are not the same thing. The kernel is the OS core running in kernel space. systemd is a regular process (PID 1) running in user space.
The notification always comes from the kernel. systemd's only role is being the last-resort parent for orphans:
| Event | Who handles it |
|---|---|
| container dies | kernel wakes containerd-shim-runc-v2 |
containerd-shim-runc-v2 dies | kernel wakes systemd |
systemd dies | kernel panic, machine is dead |
systemd is a passive adopter. containerd-shim-runc-v2 is an active guardian. systemd doesn't know or care what containerd-shim-runc-v2 is; it just calls wait() when an adopted child dies. containerd-shim-runc-v2 knows exactly what it's watching and reports the exit code back to containerd.
v1 → v2: double fork added, fewer processes
Both v1 and v2 shims hold stdio, collect exit codes with wait, and mark themselves as subreapers. v1 has no double fork, though. In v1.2.0's client.go and shim_linux.go, containerd starts containerd-shim directly as its child and Waits on it, and the shim is adopted by PID 1 only when containerd exits. So on a node still running v1, the shim sits under containerd, not under PID 1. v2 added the start step, and also changed how many shim processes you run:
v1: one containerd-shim per container
Under CRI every Pod also has a sandbox (pause) container, and openEuler's comparison of shim v1 and v2 notes that under v1 it got its own shim too.
- Pod with 3 containers → 4
containerd-shimprocesses (3 + the pause container) - 30 Pods x 2 containers average → 90
containerd-shimprocesses on one Node
v2: one containerd-shim-runc-v2 per Pod
- Pod with 3 containers → 1
containerd-shim-runc-v2process managing all three - 30 Pods → 30
containerd-shim-runc-v2processes (not 90)
Both versions talk to containerd over ttrpc, containerd's lightweight take on gRPC. Per the ttrpc README, it drops the net/http, HTTP/2 and grpc packages for a simple framing protocol, so binaries are smaller and use less memory. v2's win is the process count, not the transport. gRPC for shims only arrived as an option in containerd 1.7, and containerd's runtime v2 docs still call it experimental and recommend ttrpc.
The naming convention also made runtimes pluggable. containerd turns a runtime_type into a binary name:
io.containerd.runc.v2→containerd-shim-runc-v2→ usesrunc(default)io.containerd.runsc.v1→containerd-shim-runsc-v1→ uses gVisorio.containerd.kata.v2→containerd-shim-kata-v2→ uses Kata
Adding a runtime takes three steps: put its shim binary and runtime binary on the Node, register a handler with that runtime_type in containerd's config, and create a RuntimeClass that names the handler. A Pod that sets runtimeClassName then gets that shim. When the kernel has a bug, what keeps containers apart? walks through the gVisor version.
References
- Implementing Container Runtime Shim: runc (iximiuz) — shim daemonization, double fork walkthrough, stdio lifecycle
- Shim-shiminey Shim-shiminey (Container42) — why containerd needs shims, lifecycle detachment, live restore
- containerd runtime v2 docs (official, shim API and ttrpc)
- ttrpc README
- Containerd组件 - containerd-shim-runc-v2作用 (cnblogs)
- Zombie process (Wikipedia) — zombie vs orphan, process table entry, PID exhaustion
- Bash manual: Signals
- What are the differences between runc v1 and v2? (containerd Discussion #7407)
- Shim v1 vs v2 architecture comparison (openEuler, Kata perspective)