Debugging your Linux kernel with Ftrace

Ftrace is the official tracer of the Linux kernel. It was introduced into the kernel in 2007 (version 2.6.30). Since then, it has grown and become more flexible and powerful. This talk will give an overview of the power of ftrace to not only debug a kernel, but simply see what is happening within it. The talk will cover how to use the function and function_graph tracer, how to take advantage of more than a thousand events that are in the kernel today. It will also talk about some of the debugging tricks with using trace_printk(), tracing_off, and ftrace_dump(), as well as mention the gdb crash utility, that lets you pull the ftrace ring buffer data from a kernel core dump and be able to analyze the trace output offline.

This talk will be useful for both new and veteran developers, and even for people that are not kernel developers. This could be useful for admins that want to see how their applications are utilizing the CPUs (with the scheduler events), or how much they are page faulting (with the memory events). As ftrace is so vast, this talk wont be able to cover all of what one can accomplish with ftrace, but it will give just enough to let people know what is available for them.

Some of the key aspects to go over:

• Where to find the ftrace interface. /sys/kernel/debug/tracing

• User tools to interact with ftrace. trace-cmd and KernelShark.

• Function and function graph tracers

• Triggering a stack dump at a specific function or event, with conditionals

• Internal kernel utilities, such as trace_printk(), tracing_off() and ftrace_dump()

• Sysctrl options, like disabling tracing on a warning or on panic

• The snapshot buffer (save on conditions)

• The crash utilities trace dump to retrieve the ftrace buffers from a crashed kernel core dump

• How to have your user space applications also write into the tracing buffer.

There's much more but that's about all that could be covered in the allotted time.