When the kernel has a bug, what keeps containers apart?
Containers on the same node all use the same host kernel. When that kernel has a bug, any one container may use it to take over the whole machine, and reach the other Pods with it.
CVE-2022-0185, from 2022, is such a bug. Until the kernel is fixed, there are three options:
| Option | What it does | Against a kernel bug |
|---|---|---|
| seccomp | Limits which syscalls a container can make | Helps only if the bug is in a syscall it forbids |
| gVisor | Hands the container's syscalls to gVisor's own kernel | Syscalls never reach the host kernel |
| Kata Containers | Runs each Pod in its own small VM | Syscalls go into the VM's own kernel |
So if a node runs code you don't fully trust, such as several customers sharing one machine, seccomp alone is not enough. To keep containers away from the host kernel, use gVisor or Kata.
Separately, setting the host's user.max_user_namespaces to 0 stops containers from opening user namespaces, so an attack like CVE-2022-0185 that needs one first fails at step one (the CVE section below covers this). The next bug may not need a user namespace, though, so like seccomp this setting only works against one kind of attack.
What the kernel does when a program reads a file
The kernel is the lowest layer of the operating system. It manages the disk, memory and network, and decides what each program is allowed to do. Ordinary programs cannot touch the hardware directly; they ask the kernel, and that request is called a syscall (system call).
Type cat notes.txt in a terminal, and the kernel does it in steps:
openat: opennotes.txtread: read the file's contentswrite: print them to the screen
cat makes these syscalls, and the kernel carries out each one. So if the kernel's code has a bug, any program that can make the matching syscall may trigger it.
A container has no kernel of its own
A VM is a whole computer simulated in software, with its own kernel. A container is different. runc is the program that actually creates containers, and it is the most common default in Kubernetes. A container run by runc is just an ordinary process on the host, using the host's kernel.
A container looks like a separate machine because the host's kernel decides its boundaries:
- Namespaces decide what it can see. Run
psinside a container and it lists only that container's processes. - cgroups decide how much CPU and memory it can use.
The kernel decides the boundaries, and every container's syscalls go into that same kernel. If the kernel itself has a bug, a program in a container may use a syscall to get around the boundaries entirely.
CVE-2022-0185
CVE-2022-0185 is a Linux kernel bug made public in January 2022. It affects kernels from 5.1 on and is fixed in 5.4.173, 5.10.93, 5.15.16 and 5.16.2. As CrowdStrike's analysis explains, when legacy_parse_param, the kernel code that handles mount parameters, gets a parameter of 4095 bytes or more, it writes where it should not in kernel memory, and an attacker can use that to become root.
From inside a container, the attack takes three steps:
- Call
unshareto open a new user namespace. The kernel treats the program as root inside it and gives it CAP_SYS_ADMIN, the permission needed to mount filesystems. An ordinary container does not have it. - With CAP_SYS_ADMIN, call
fsconfigwith a parameter of 4095 bytes or more, triggering the bug. - Overwrite kernel memory and become root on the whole host.
Step 1 fails if the Pod has seccomp applied (Kubernetes applies none by default), if runc switches the root directory with chroot instead of its default pivot_root, or if the host sets user.max_user_namespaces to 0.
The most direct fix is to upgrade the kernel. But the next bug will come, and until it is fixed, the three options below are what is left.
seccomp: limit which syscalls can be made
seccomp is a kernel feature that gives a program an allow list of syscalls and rejects everything else. Kubernetes has containerd or CRI-O create containers on each node, and both ship a default list, which a Pod selects with RuntimeDefault:
spec:
securityContext:
seccompProfile:
type: RuntimeDefaultCrowdStrike added this and tried again. Step 1's unshare failed with "Operation not permitted", and the attack stopped there.
But Kubernetes does not apply this list on its own. A Pod without seccompProfile runs as Unconfined, which filters nothing. To apply it to every Pod by default, start the kubelet (the Kubernetes program on each node that starts Pods) with the --seccomp-default flag, which Kubernetes's seccomp docs list as stable since 1.27.
seccomp has a limit. The list always allows syscalls like read that every program needs. If the next bug is in one of those, seccomp does nothing, because allowed syscalls still go into the host kernel.
gVisor: put another kernel in between
gVisor is a Google project. It puts another kernel, called the Sentry, between the container and the host kernel. The Sentry is written in Go and is an ordinary process on the host. Syscalls from programs in the container go to the Sentry, and the Sentry's own code handles them. gVisor's architecture docs say "gVisor never passes through any system call to the host".
Normal container: program → host kernel
gVisor: program → Sentry → host kernel (only the few calls the Sentry itself needs)
Back to CVE-2022-0185: unshare and fsconfig are handled by the Sentry, or rejected with an error if it does not implement them. Either way they never reach the host kernel, so the buggy legacy_parse_param never runs.
The Sentry is a process too, so it still asks the host kernel for some things, such as timers and signals. As gVisor's security docs explain, those requests are for the Sentry's own work; the program in the container cannot choose which syscall the Sentry makes or with what arguments.
Reading and writing files used to go through another process called the Gofer: the Sentry could not open files on the host itself and asked the Gofer to open them. According to gVisor's announcement, since June 2023 gVisor uses directfs by default, letting the Sentry open files in the container directly, which is faster. It also means the Sentry needs more permission on the host, and gVisor's help text for the directfs setting says "Sentry runs with higher privileges".
gVisor's limits:
- Kernel features the Sentry does not implement are not available to programs.
- Every syscall goes through the Sentry first, so programs that make syscalls very often and do little work for each slow down noticeably; the official example is small Redis operations. Networking and file I/O are also slower for now, mainly, the performance docs say, because the implementation still needs work.
Kata Containers: a small VM for each Pod
Kata Containers starts a lightweight VM for each Pod, and the Pod's containers run inside it. The VM has its own kernel, called the guest kernel. The program on the host that starts VMs is called a hypervisor; Kata's architecture docs list several it supports, including QEMU and Cloud Hypervisor.
Normal container: program → host kernel
Kata: program → guest kernel (this Pod's own) → hypervisor → host kernel
Back to CVE-2022-0185: unshare and fsconfig go into this Pod's own guest kernel. Even if the guest kernel has the same bug, the attacker only gets this VM; the host and other Pods are still outside it. To go further, they need a bug in the hypervisor too. Kata's documentation calls the VM a "second layer of defence".
Kata's limit is one more kernel running for every Pod, and a hypervisor installed on the node.
The three side by side
| Who handles the program's syscalls | What an attacker must get past to use a host kernel bug | |
|---|---|---|
| runc (default) | host kernel | nothing |
| runc with seccomp | host kernel, only syscalls the list allows | the bug must be in a syscall the list allows |
| gVisor | the Sentry | break the Sentry first, then use the small set of host syscalls the Sentry may make (with directfs on by default, these include opening files in the container) |
| Kata | this Pod's guest kernel | take over the guest kernel first, then break out through the hypervisor |
I found no source comparing their security under the same conditions, so this table does not rank them. gVisor's documentation points out that a VM has its own weak spot. When a program in a VM uses a network card or a disk, code on the hypervisor side emulates those devices, and if that code has a bug, an attacker can reach the host from the VM. On that basis, gVisor's documentation warns that using a VM does not by itself make a system safer than gVisor.
Switching to gVisor or Kata in Kubernetes
containerd does not call runc directly. It first starts a small program called a shim, containerd-shim-runc-v2 for normal containers, which calls runc and then stays to report the container's status. Details are in Inside containerd: How shim and runc Actually Work.
gVisor and Kata each ship their own shim:
- gVisor's is
containerd-shim-runsc-v1, which callsrunsc, gVisor's replacement for runc that starts the Sentry. - Kata's is
containerd-shim-kata-v2, which is itself the Kata runtime and starts the VM through the hypervisor.
So the node needs the new shim installed, and for gVisor, runsc as well. After that, the gVisor setup from its containerd quick start takes three steps.
- Add a runtime handler to containerd's config, which gives the option "run with gVisor" a name, here
runsc. The lineruntime_type = "io.containerd.runsc.v1"below it names the shim to use, and containerd maps that value to the executablecontainerd-shim-runsc-v1by its naming rule.# /etc/containerd/config.toml version = 2 # without it, containerd 1.x reads the file as version 1 and skips the section below [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc] runtime_type = "io.containerd.runsc.v1" - Create a RuntimeClass, a Kubernetes object that maps a name (here
gvisor) to a handler on the node (runsc). - Set
runtimeClassNamein the Pod spec to that RuntimeClass.
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: gvisor
handler: runsc
---
# in the Pod spec
spec:
runtimeClassName: gvisorThe image stays the same, and the Pod spec gains only the runtimeClassName line. Per the RuntimeClass docs, a Pod without runtimeClassName uses the node's default handler and runs on runc as before.
References
- CrowdStrike: CVE-2022-0185 Kubernetes Container Escape Using Linux Kernel Exploit
- K8s docs: Restrict a Container's Syscalls with seccomp
- gVisor: Introduction
- gVisor: Security model
- gVisor: Performance guide
- gVisor: Directfs (2023)
- clone(2) man page
- gVisor: containerd quick start
- Kata Containers: Architecture
- K8s docs: RuntimeClass