SystemTap

SystemTap is a dynamic tracing mechanism with many similarities to the Solaris DTrace tool. It allows the placement of probes anywhere in the kernel and supports a C-like scripting language for specifying what happens when those probes are encountered.

Forecast: the core technologies used by SystemTap (kprobes and relay) are already part of the mainline kernel, and most distributors provide SystemTap packages now. SystemTap is painful to set up and use, though, especially when compared to DTrace. What this tool needs is some significant usability work on the part of its developers and the distributors. The infrastructure is in place, but the task of finishing the job has not been done.

Note that there is an increasing amount of competition from other tracing frameworks as some developers have grown tired of waiting for SystemTap and have begun working on alternatives.

For more information:

• On DTrace envy (August, 2007)
• More DTrace envy (July, 2008)
• A SystemTap update (January, 2009)

Kernel markers

The kernel markers project aims to create a low-impact means for the insertion of static trace points into the kernel source. These trace points could be used with SystemTap, LTTng, or any other tracing mechanism.

Forecast: This patch has been merged for 2.6.24.

For more information:

• Kernel markers (August, 2007)

utrace

The utrace subsystem is a kernel infrastructure useful for tracing (and controlling) the behavior of processes. The immediate application is the replacement of the old, much-criticized ptrace() interface, but there is much more which can be done with it.  In particular, utrace is seen by some as an important component of a solution for user-space dynamic tracing.

Forecast: utrace has been in and out of the -mm tree; it remains a controversial development.  There is no doubt that many of the things that utrace is trying to support - a cleanup of ptrace() support, perhaps the eventual creation of a better system call for tracing, support for dynamic user-space tracing - are useful and desired, but there is no consensus that the correct solution has been found.

For more information:

• Introducing utrace (March, 2007)
• The return of utrace (March, 2009)
• Another mainline push for utrace (December, 2009)
• Uprobes: not quite there yet (January, 2010)
• Back to the drawing board for utrace? (January, 2010)
• Replacing ptrace() (January, 2010)

KGDB

Linux has long lacked support for an interactive kernel debugger - at least, in the mainline kernel. There is, however, now a patch set which aims at providing comprehensive KGDB support for most architectures. Debugging over serial lines and network connections is supported.

Forecast: There is a certain amount of demand for this feature, but Linus has famously opposed the use of interactive debuggers for years. But activity around KGDB has picked up, and it is now being fixed up by the core x86 developers. As a result, a minimal version of KGDB was merged for 2.6.26.  Some important features (such as debugging over the network) remain outside, though.

This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.