Step-by-Step Tutorial: Guide to install systemtap in Linux

SystemTap is a GPLv2 license system wide tool that allows you to gather tracing and profiling data from a running Linux system. The user writes a stap script, which you can then compile into a Linux kernel module to link it against the same kernel source it is going to run under.

The script sets events and handlers, which are called by the kernel module on the specified events triggering. For this, it uses the kprobes and uprobes (if available) interfaces in the kernel.

Understanding systemtap

• Systemtap connects to the Linux kernel and monitors for available events, which are exposed through the kprobes kernel facility
• Based on an event, the kernel can run a handler which is executed as a sub routine
• The event and handler together are refers as probe
• Stap is doing its work by running scripts. To work, these scripts you need to compile the stap scripts in the kernel and start it as a kernel module. The kernel module next will do its work and after doing so it will unload
• The stap command will run the scripts and do the compilation

Monitoring kernel events

Events that are available with ftrace can be accessed with systemtap

• Kernel functions
  • probe.kernel.function("function-name")
• Kernel system calls
  • probe.syscall."syscall-name"
• Kernel tracepoints
  • probe.kernel.trace("tracepoint")

Install systemtap

Systemtap installs packages that normally you don't want to have on a production system, it brings serious alterations to the system and for that reason should be used in test environments only.

The following packages are the mandatory pre-requisite

• kernel-devel
• kernel-debuginfo
• kernel-common-debuginfo
• systemtap
• systemtap-debuginfo

You can create your own repository for this purpose to download all the required dependencies. Before starting with the installation check the active kernel version.

# uname -r
3.10.0-862.14.4.el7.x86_64

# rpm -qa | grep kernel
kernel-headers-3.10.0-862.14.4.el7.x86_64
kernel-tools-libs-3.10.0-862.14.4.el7.x86_64
kernel-3.10.0-862.14.4.el7.x86_64

So now we know we need kernel-devel, kernel-debuginfo and kernel-common-debuginfo with 3.10.0-862.14.4.el7.x86_64 version.

# yum -y install kernel-debuginfo*3.10.0-862.14.4* kernel-devel-*3.10.0-862.14.4* kernel-common-debuginfo*3.10.0-862.14.4*

Next it is time to install systemtap and it's pre-requisite rpms

# yum -y install systemtap systemtap-debuginfo

Lastly after installation of all the systemtap and pre-requisite rpms, reboot the node for the changes to take affect.

So now your node is all setup to execute stap.

Initial testing

If you are currently using the kernel you intend to probe with SystemTap, you can immediately test whether the deployment was successful. If not, you restart the system and load the appropriate kernel.

To start the test, run the following command:

# stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}'

This command instructs SystemTap to print read performed and then exit properly once a virtual file system read is detected.  If the SystemTap deployment was successful, it prints output similar to the following:

# stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}'
Pass 1: parsed user script and 479 library scripts using 248000virt/49976res/345 6shr/46528data kb, in 770usr/30sys/976real ms.

Pass 2: analyzed script: 1 probe, 1 function, 7 embeds, 0 globals using 408460virt/172252res/1540shr/206988data kb, in 3590usr/1040sys/16653real ms.
Pass 3: translated to C into "/tmp/stapzKM0yC/stap_b91faccbe0f7c63a74a89a2ddb3f15c2_2696_src.c" using 408460virt/173248res/2484shr/206988data kb, in 20usr/0sys/48real ms.
Pass 4: compiled C into "stap_b91faccbe0f7c63a74a89a2ddb3f15c2_2696.ko" in 1570usr/330sys/3301real ms.
Pass 5: starting run.
read performed
Pass 5: run completed in 10usr/20sys/322real ms.

Running SystemTap scripts

• Default scripts are installed with the SystemTap package available inside /usr/share/systemtap/examples/
• Scripts are prepared with the stap command
  • Check the script syntax
  • Expand the script so that it includes probes and function calls
  • Translate the script language to C
  • Compile that as a kernel module
• Once compiled, use staprun to run the script
  • Add -v for more verbosity

Compiling SystemTap scripts

• Kernel modules compiled with stap may be exported so that they can run on a different system
• Normally when writing systemtap scripts, 5 runs are processed
• Use stap -p 4 -v ~/tmp/mymodule.stp to run the compilation only
• The -p 4 option will stop after running pass 4 of the stap command, which is when the compilation happens

Below is a simple stap script which will print hello world

# cat hello-world.stp
#!/use/bin/env stap

probe begin
{
    printf ("hello world\n")
    exit ()
}

Let us execute this script with verbosity

# stap -v hello-world.stp
Pass 1: parsed user script and 479 library scripts using 247996virt/49972res/3448shr/46524data kb, in 810usr/30sys/996real ms.
Pass 2: analyzed script: 1 probe, 1 function, 0 embeds, 0 globals using 249316virt/51552res/3748shr/47844data kb, in 20usr/0sys/16real ms.
Pass 3: using cached /root/.systemtap/cache/b0/stap_b013d9850b952b9fa68246bba7df5605_1013.c
Pass 4: using cached /root/.systemtap/cache/b0/stap_b013d9850b952b9fa68246bba7df5605_1013.ko
Pass 5: starting run.
hello world
Pass 5: run completed in 0usr/10sys/298real ms.

Here as we see there are 5 runs of the execution and at the 4th run we get the kernel module for our stap script.

Now we can stop the script execution at 4th run when the kernel module gets created

# stap -v -p 4 hello-world.stp
Pass 1: parsed user script and 479 library scripts using 247868virt/49964res/3448shr/46396data kb, in 820usr/20sys/852real ms.
Pass 2: analyzed script: 1 probe, 1 function, 0 embeds, 0 globals using 249188virt/51544res/3748shr/47716data kb, in 10usr/0sys/13real ms.
/root/.systemtap/cache/b0/stap_b013d9850b952b9fa68246bba7df5605_1013.ko
Pass 3: using cached /root/.systemtap/cache/b0/stap_b013d9850b952b9fa68246bba7df5605_1013.c
Pass 4: using cached /root/.systemtap/cache/b0/stap_b013d9850b952b9fa68246bba7df5605_1013.ko

Here as expected now the script execution stopped at 4th run and now we can access our kernel module for the provided stap script

# ls -l /root/.systemtap/cache/b0/stap_b013d9850b952b9fa68246bba7df5605_1013.ko
-rw-r--r-- 1 root root 86232 Dec 2 21:09 /root/.systemtap/cache/b0/stap_b013d9850b952b9fa68246bba7df5605_1013.ko

To run a SystemTap module that was created somewhere else, you need to install systemtap-runtime, which provides the staprun command. So now we can use staprun to execution this kernel module

# staprun /root/.systemtap/cache/b0/stap_b013d9850b952b9fa68246bba7df5605_1013.ko
hello world

So the script is working and we get a hello world output.

Example script

This is a sample stap script which will help you understand if the process received a signal 15 (SIGTERM)

# cat /tmp/getstatus.stp
#!/usr/bin/env stap

probe signal.send {
  time_ns = gettimeofday_ns();
  if (pid_name == @1) {
     if (sig_name == "SIGTERM") {
        printf("%ld: %s(pid: %d) received a %s signal sent by %s(%d)\n", time_ns, pid_name, sig_pid, sig_name, execname(), pid())
        exit()
     }
  }
}

So let us execute the script and monitor dhcpd process

# stap /tmp/getstatus.stp dhcpd

On another terminal I will stop the dhcpd process using signal 15 (SIGTERM) and monitor the stap status

# stap /tmp/getstatus.stp dhcpd
1543762612968405347: dhcpd(pid: 4201) received a SIGTERM signal sent by bash(3517)

As we see, it prints a message as expected. So our configuration is working as expected.

Lastly I hope this article to setup and install SystemTap on Linux was helpful. So, let me know your suggestions and feedback using the comment section.