Not breaking userspace: the evolving Linux ABI

"Don't break userspace" is famously one of the core tenets of Linux kernel development. The rule makes it possible for distros to upgrade the kernel independently of userspace applications.  But what does the phrase actually mean in 2022?  What portions of the kernel's interface are guaranteed stable and are safe for applications to rely on?

Kernel contributors try to make only backward-compatible changes to the procfs and sysfs filesystems. These virtual filesystems (VFS) constitute the primary applications binary interface (ABI).  System calls in a C library like glibc send commands to and receive information from the kernel via the VFS. Linux userspace applications can then rely on the syscalls to interact with the OS or access the VFS directly.

Linux OS's have many artifacts besides procfs and sysfs, however. Which of them are part of the stable ABI? The kernel's documentation calls out the fact that kernel configuration parameters are not stable, but what about devicetree formats, the dmesg log, the BPF validator interface, /dev files, kernel command-line parameters, tracepoints and filesystem metadata? Changes to not explicitly stable parameters have often been the source of controversy.

The difficulty of adding features and fixing bugs without changing the stable ABI is universal in software development. Sometimes there is a stark choice between implementing an upstream fix and "breaking" the library's users. When inevitably the kernel ABI is modified for bug fixes, C-library developers try to balance support for the changes with customer stability expectations.  For example, a recent fork of libpthread with an ABI break was created by the realtime Linux community in order to fix problems in time-critical systems like robotics.  In 2021, the kernel added a "printk indexing" feature that will make it easier for applications to parse error messages in the dmesg log.  The presentation should equip attendees to better understand the Linux ABI and feel better prepared to deal with changes to them that affect their own projects.