When the kernel has a bug, what keeps containers apart?

A Kubernetes node often runs several Pods at once, from different teams or even different customers. If one of those Pods runs malicious code, can it reach the other Pods? Can it take over the whole machine?

The answer depends on how containers relate to the kernel.

What the kernel does when a program reads a file

The kernel is the lowest layer of the operating system. It manages the hardware (disk, memory, network) and decides what each program is allowed to do. Ordinary programs cannot touch the hardware directly. To get anything done they ask the kernel, and that request is called a syscall (system call).

Take cat notes.txt on Linux. cat cannot read the disk by itself. It does this:

1. Calls the openat syscall to ask the kernel to open notes.txt
2. Calls read to ask the kernel for the file's contents
3. Calls write to ask the kernel to print them to the screen

The kernel does the real work at every step. So if the kernel's code has a bug, any program that can make the matching syscall has a chance to trigger it.

A container has no kernel of its own

Containers are often compared to VMs (virtual machines), but they are built differently. A VM is a whole computer simulated in software, running its own kernel. runc is the program that actually creates containers, and it is the most common default in Kubernetes. A container run by runc is just an ordinary process on the host, and it uses the host's kernel.

A container looks like a separate machine because the host's kernel draws boundaries around it:

• Namespaces decide what the process can see. Run ps inside a container and it lists only that container's processes.
• cgroups decide how much CPU and memory the process can use.

Both are kernel features. The kernel draws the boundaries, and every syscall from every container on the node goes into that same kernel. If the kernel itself has a bug, a program in a container may trigger it through a syscall and get around the boundaries the kernel drew.

A real example: CVE-2022-0185

CVE-2022-0185 is a Linux kernel bug made public in January 2022, present in kernels from 5.1-rc1 on. It is in the function legacy_parse_param, which handles parameters passed in when a filesystem is mounted. When a parameter is 4095 bytes or longer, the kernel writes to places in its memory it should not. CrowdStrike's analysis says an attacker can use it to go from an ordinary user to root, and the boundaries namespaces draw do not help (CrowdStrike).

To get a parameter into legacy_parse_param, a program has to call the fsconfig syscall, and fsconfig requires CAP_SYS_ADMIN.

CAP_SYS_ADMIN is a capability. Linux splits root's powers into a few dozen capabilities, each covering one kind of task, and CAP_SYS_ADMIN covers system administration such as mounting filesystems. An ordinary container does not have it, so it should not be able to call fsconfig.

A program in a container has another route, though: a user namespace. By calling the unshare syscall, it can open a new user namespace. The kernel treats the program as root inside that namespace and gives it CAP_SYS_ADMIN there. That power only counts inside the new namespace, but mounting a filesystem inside that namespace is all the program wants to do, so the kernel's permission check lets fsconfig through. After that, the host's one and only kernel handles fsconfig as usual and runs the buggy legacy_parse_param.

The attack, in order:

1. Inside the container, call unshare to open a new user namespace and get CAP_SYS_ADMIN in it
2. Call fsconfig with a parameter of 4095 bytes or more, triggering the bug in legacy_parse_param
3. Overwrite kernel memory and become root on the whole host

Step 1 only works if three conditions all hold:

• The Pod does not restrict syscalls. CrowdStrike points out that Kubernetes applies no seccomp, AppArmor or SELinux restriction to a Pod by default, so a program in a Pod can call unshare directly.
• The container's root directory was switched the way runc does it by default. The / a container sees is really some folder on the host, and runc uses pivot_root to make that folder the container's root. If the older chroot is used instead, step 1 fails: the clone(2) man page says a program under a root directory changed by chroot cannot open a new user namespace. runc uses chroot only when run with --no-pivot.
• The host allows user namespaces. Linux has a system setting, user.max_user_namespaces, that limits how many user namespaces can be opened. If an administrator sets it to 0, no program can open one, and step 1 fails too. The downside is that programs that need user namespaces stop working as well, such as containers run without root (rootless containers).

The most direct fix is to upgrade the kernel, and that is CrowdStrike's first recommendation. But kernel bugs keep turning up, and there has to be a defence until each one is fixed. seccomp keeps a container from calling certain syscalls. gVisor and Kata keep a container's syscalls from reaching the host's kernel at all.

seccomp: limit which syscalls a container can make

seccomp is a Linux kernel feature that gives a program an allow list of syscalls and rejects everything else. On a node, Kubernetes usually hands container creation to one of two programs, containerd or CRI-O. Each ships a default list, which a Pod selects with RuntimeDefault:

spec:
  securityContext:
    seccompProfile:
      type: RuntimeDefault

CrowdStrike added this to a Pod and ran the attack again. Step 1's unshare failed with "Operation not permitted". The program never got CAP_SYS_ADMIN, so it could not call fsconfig afterwards.

Kubernetes does not apply this list on its own, though. A Pod that sets no seccompProfile runs as Unconfined, which filters nothing. To make every Pod use RuntimeDefault unless it says otherwise, start the kubelet (the Kubernetes program on each node that starts Pods) with the --seccomp-default flag, which has been stable since Kubernetes 1.27 (Kubernetes seccomp docs).

seccomp has a limit. The list always allows syscalls like read that every program needs, and those still go straight into the host's kernel. If the next bug sits in one of them, seccomp does not help.

gVisor: put a separately written kernel in between

gVisor is a Google project. It puts a second kernel between the container and the host's kernel. This kernel is called the Sentry; it is written in Go and runs as an ordinary process on the host. Syscalls from programs in the container go to the Sentry, and the Sentry's own code handles them. gVisor's architecture guide says "gVisor never passes through any system call to the host": no syscall is handed on to the host's kernel as it is (gVisor architecture).

Normal container:  program → host kernel
gVisor:            program → Sentry → host kernel (only the few calls the Sentry itself needs)

Back to CVE-2022-0185: the program's unshare and fsconfig are handled by the Sentry, or rejected with an error if the Sentry does not implement them. Either way they never reach the host's kernel, so the buggy legacy_parse_param never runs.

The Sentry is itself a process on the host, so it still has to ask the host's kernel for some things, such as timers and signals. Those requests are made for the Sentry's own work, and the program in the container cannot choose which syscall the Sentry makes or with what arguments. gVisor's security page lists the small set of host syscalls the Sentry may use (gVisor security).

As the security page describes it, the Sentry cannot open files on the host itself. It asks a separate process called the Gofer to open them and only receives the opened files. In June 2023 gVisor changed the default so the Sentry opens files in the container directly; this mode is called directfs. Reading and writing files no longer needs a trip to the Gofer each time, so it is faster. It also means the Sentry process needs permission on the host to open files directly, and gVisor's help text for the setting says "Sentry runs with higher privileges" (gVisor announcement).

gVisor has costs too:

• Kernel features the Sentry does not implement are not available to programs under gVisor.
• Every syscall goes to the Sentry first, so programs that make syscalls very often and do little work for each slow down noticeably; gVisor's performance guide uses small Redis operations as its example. The more work a program does between syscalls, the smaller the effect. The guide also says networking and file I/O are currently slow mainly because the implementation still needs improvement (gVisor performance).

Kata Containers: a small VM for each Pod

Kata Containers starts a lightweight VM for each Pod, and all the Pod's containers run inside it. The VM has its own kernel, called the guest kernel. The program on the host that starts and runs VMs is called a hypervisor, and Kata supports several, including QEMU and Cloud Hypervisor (Kata architecture).

Normal container:  program → host kernel
Kata:              program → guest kernel (this Pod's own) → hypervisor → host kernel

Back to CVE-2022-0185: unshare and fsconfig go into this Pod's own guest kernel. If the guest kernel has the same bug, the attacker gets control of this VM, while the host and other Pods are still outside it. To go further, they would need a bug in the hypervisor too. Kata's documentation calls the VM layer a "second layer of defence".

The cost is one more kernel running for every Pod, and a hypervisor installed on the node.

The approaches side by side

I found no source that compares the security of these approaches under the same conditions, so this table does not rank them. gVisor's security page does point out that when a program in a VM uses a network card or a disk, code on the hypervisor side emulates those devices for it, and that code is the barrier the Kata row asks an attacker to break. Like the Sentry, the device-emulation code sits between the program in the container and the host kernel, and if it has a bug, an attacker can reach the host from the VM. So gVisor's documentation holds that using a VM does not by itself make a system safer than gVisor.

Switching to gVisor or Kata in Kubernetes

containerd does not call runc directly when it creates a container. It first starts a small program called a shim, containerd-shim-runc-v2 for normal containers, and the shim calls runc. Once the container is running, the shim stays and reports the container's status. How the shim works is in Inside containerd: How shim and runc Actually Work.

gVisor and Kata each ship their own shim. gVisor's is containerd-shim-runsc-v1, which then calls runsc, gVisor's replacement for runc that starts the Sentry. Kata's is containerd-shim-kata-v2, which is itself the Kata runtime and starts the VM through the hypervisor. So to replace runc, the node needs the new shim installed, and for gVisor, runsc as well.

With those installed, the gVisor setup from its containerd quick start takes three steps.

1. Add a runtime handler to containerd's config, which gives the option "run with gVisor" a name, here runsc. The line runtime_type = "io.containerd.runsc.v1" below it names the shim to use, and containerd maps that value to the executable containerd-shim-runsc-v1 by its naming rule.

   # /etc/containerd/config.toml
   version = 2  # without it, containerd 1.x reads the file as version 1 and skips the section below
   [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc]
     runtime_type = "io.containerd.runsc.v1"

2. Create a RuntimeClass. A RuntimeClass is a Kubernetes object that maps a name (here gvisor) to a handler on the node (runsc).

3. Set runtimeClassName in the Pod spec to that RuntimeClass.

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc
---
# in the Pod spec
spec:
  runtimeClassName: gvisor

The image stays the same, and the Pod spec gains only the runtimeClassName line. A Pod without runtimeClassName uses the node's default handler and runs on runc as before (K8s RuntimeClass).

References

• CrowdStrike: CVE-2022-0185 Kubernetes Container Escape Using Linux Kernel Exploit
• K8s docs: Restrict a Container's Syscalls with seccomp
• gVisor: Introduction
• gVisor: Security model
• gVisor: Performance guide
• gVisor: Directfs (2023)
• clone(2) man page
• gVisor: containerd quick start
• Kata Containers: Architecture
• K8s docs: RuntimeClass