Kubernetes and the Dragons in Linux Kernel vs. Userspace Tools

Imagine performing a routine Linux OS update on your Kubernetes cluster nodes. Boom—everything breaks, and suddenly you can’t even SSH into the nodes. You instantly know it’s going to be a “fun” day. The culprit? A subtle iptables version incompatibility lurking beneath the surface. As Kubernetes system components increasingly package low-level system tools, a new set of compatibility challenges has emerged. This talk explores three such “dragons” that Kubernetes engineers face when kernel compatibility issues crop up in containerized environments: modprobe, iptables, and ipset.

Each of these system tools, developed long before containers were widespread, brings unique and often hidden compatibility issues. First, we’ll explore modprobe, where Debian-based container images ship with gzip-compressed kernel modules incompatible with the xz compression method used by Kubernetes components, making it impossible for certain containers to load kernel modules. Next, we’ll dive into iptables, the tool at the heart of Linux and Kubernetes networking, where incompatibilities between versions can break critical functions and complicate rule management across components and host levels. Finally, we’ll look at ipset, a tool that fails if its version is older than the kernel itself, leading to unexpected issues during routine kernel updates.

We’ll share how these challenges affect Kubernetes stability, some of the troubleshooting paths we took, and what we did to handle these problems. Join us to understand the tricky relationship between Kubernetes, the Linux kernel, and the low-level tools we rely on—and learn how to prepare for the next compatibility dragon lurking in your stack.