Getting Around to It: Deferred Work Mechanisms in the Linux Kernel

The Linux kernel has multiple methods for scheduling delayed work, notably tasklets, workqueues, softirqs, waitqueues and irq_work.  Why are there so many kinds of deferred work, and how do they differ?  How might system administrators and developers detect misbehavior of related kernel threads and improve their performance?

A major distinction in deferred work types is between callbacks whose invocation is triggered by an event such as a interrupt, as opposed to already active tasks which must wait due to resource constraints.  A second distinction is between jobs of predictable duration that don't block versus computationally intensive tasks of long duration.  The former are well suited to tasklets, while the latter are best serviced by workqueues.  Subsystems including crypto, audio and graphics make heavy use of tasklets.  Kworkers are long-lived kernel threads which execute the tasks scheduled via workqueues.  Somewhat surprisely, a given kworker may execute a variety of unrelated workloads related to (for example) filesystems, graphics and memory management. Recent kernel releases include improvements to workqueue performance as well as adding new tools to monitor their behavior.

Softirqs are an unloved legacy callback for hardware interrupt events.  They come in block, network, timer, RCU and tasklet flavors.  Softirqs may follow directly after the associated hard IRQ, or they may be executed by the somewhat notorious per-CPU ksoftirqd thread, which realtime Linux chief Thomas Gleixner has termed "the new big kernel lock."  Like kworkers, ksoftirqd services a variety of unrelated tasks with differing latency needs.  Sysadmins may configure systems to move RCU and network callbacks out of ksoftirqd.

Common signs of system inefficiency include deadline misses due to bursts of heavy network traffic, kworker watchdog timeouts due to lock contention, or RCU stalls due to delayed execution of ksoftirqd.  Delaying ksoftirqd execution can also result in the timer softirq not running promptly, causing a knock-on malfunction of the scheduler. Long-running efforts to minimize problems caused by ksoftirqd and improve the reliability of timers may soon reach the kernel mainline.  The presentation will demonstrate how to employ the libbpf scripts and the general-purpose and simple bpftrace tool to keep watch on a variety of kernel internals.  Attendees will learn methods to improve performance of workqueues and softirqs, including tasklets.