Analyze Linux Performance with perf and debuginfod

perf is the standard Linux performance analysis tool for counting CPU events, sampling hot functions, collecting call graphs, and tracing system activity. This guide walks through a repeatable workflow on a generic Linux host: install perf, verify kernel support, profile a CPU-bound sample program with perf stat and perf record, interpret perf report, improve symbols with debuginfod, apply practical incident-response workflows, and troubleshoot incomplete results.

The commands apply across distributions. Package names and debuginfod defaults differ; the measurement workflow stays the same.

Understand what perf can analyze

perf uses the kernel perf_events interface to observe running software. In practice you can:

• Count CPU and software events such as cycles, instructions, and page faults
• Profile one command, one process, or the whole system
• Rank hot functions from periodic samples
• Collect user-space and kernel call graphs when unwind data is available
• List tracepoints, kprobes, and uprobes for targeted recordings
• Annotate assembly or source lines when debug information is present

perf tells you where time or events accumulated. It does not automatically explain every business-logic bottleneck—you still interpret the hotspot in context.

perf top ranks hot functions and symbols inside processes—it does not initially tell you which process to choose. When the target PID is unknown, start with top or ps -eo pid,comm,%cpu --sort=-%cpu | head, then attach perf top or perf record to the selected PID.

Prerequisites

• A Linux system with perf_events support in the running kernel
• Permission to install packages and, for some recordings, sudo command access
• A compiler if you build the sample workload locally (gcc on RPM-based systems, build-essential on Debian and Ubuntu)
• binutils for readelf in the debuginfod example (readelf is not guaranteed on a minimal install)
• Outbound HTTPS if you use a remote debuginfod server

Install perf and check system support

Install the distribution package that matches your running kernel where required.

Rocky Linux and RHEL install perf from AppStream with the dnf command:

sudo dnf install perf elfutils-debuginfod-client binutils

Debian refreshes package metadata and installs through apt command:

sudo apt update
sudo apt install linux-perf debuginfod binutils

The linux-perf package already pulls in the debuginfod client library. Install the separate debuginfod package when you need the debuginfod-find command used later in this guide.

Ubuntu:

sudo apt update
sudo apt install linux-tools-common "linux-tools-$(uname -r)" debuginfod binutils

The kernel-matched linux-tools-$(uname -r) package is preferable because a host can run an HWE, cloud, low-latency, or vendor-specific kernel that linux-tools-generic does not cover. When the matched package is unavailable, try the meta package:

sudo apt install linux-tools-generic

openSUSE and SLES:

sudo zypper install perf

Confirm the debuginfod client package name on your release before you rely on debuginfod-find:

zypper search debuginfod

On many openSUSE and SLE systems the client package is debuginfod or debuginfod-client.

Confirm the tool version and running kernel:

perf version

Sample output:

perf version 6.12.0-211.34.1.el10_2.x86_64

uname -r

Sample output:

6.12.0-211.16.1.el10_2.0.1.x86_64

The perf user-space build should match the running kernel closely on distributions that ship kernel-specific tools. A small mismatch may still work, but after a kernel update reinstall the matching perf or linux-tools package if recordings fail or events disappear.

List available events:

perf list | head -20

Sample output:

msr/smi/                                           [Kernel PMU event]
  msr/tsc/                                           [Kernel PMU event]
  alarmtimer:alarmtimer_cancel                       [Tracepoint event]
  alarmtimer:alarmtimer_fired                        [Tracepoint event]
  block:block_bio_queue                              [Tracepoint event]

Event availability depends on CPU model, architecture, kernel configuration, and whether the host is bare metal or a virtual machine. VMs often expose fewer hardware performance counters than physical servers.

Run a quick access check:

perf stat true

Sample output:

Performance counter stats for 'true':

              1.232 msec task-clock        #    0.998 CPUs utilized
                  1      context-switches  #    0.812 K/sec
                  0      cpu-migrations    #    0.000 /sec
                 50      page-faults       #   40.584 K/sec
       <not supported>      cycles
       <not supported>      instructions

       0.001234567 seconds time elapsed

On the tested VM, hardware cycles and instructions reported <not supported> while software counters still worked. That is common in virtualized lab hosts; bare-metal systems usually expose fuller PMU counters. When hardware PMU events are unavailable, prefer software sampling events such as cpu-clock or task-clock for perf record and perf stat.

Check current profiling restrictions:

cat /proc/sys/kernel/perf_event_paranoid

cat /proc/sys/kernel/kptr_restrict

Sample output:

2
1

perf_event_paranoid controls how much profiling unprivileged users can perform. kptr_restrict limits exposure of kernel pointer values in /proc and reports. Prefer targeted sudo perf recordings over permanently lowering these values on multi-user systems.

Prepare a repeatable CPU-bound workload

The examples below use a small recursive Fibonacci program that accepts command-line arguments so you can keep the run long enough to collect useful samples. The defaults target roughly five seconds on a typical lab CPU; increase the Fibonacci index or repetition count when captures finish too quickly.

Save the source as perf-cpu-burn.c:

#include <stdio.h>
#include <stdlib.h>

static volatile unsigned long sink;

__attribute__((noinline))
static unsigned long fib(unsigned n)
{
    if (n < 2)
        return n;
    return fib(n - 1) + fib(n - 2);
}

int main(int argc, char **argv)
{
    unsigned n = 37;
    int reps = 50;
    unsigned long total = 0;

    if (argc > 1)
        n = (unsigned)atoi(argv[1]);
    if (argc > 2)
        reps = atoi(argv[2]);

    for (int i = 0; i < reps; i++)
        total += fib(n);

    sink = total;
    printf("done: %lu\n", total);
    return 0;
}

The noinline attribute keeps fib visible as a hotspot instead of being inlined away by the optimizer.

Compile once with DWARF debug information and retained frame pointers. The examples use an optimized binary with realistic generated code while still allowing source annotation and both frame-pointer and DWARF call-graph tests:

gcc -O2 -g -fno-omit-frame-pointer -o perf-cpu-burn perf-cpu-burn.c

Run it once to confirm output and duration:

time ./perf-cpu-burn

Sample output:

done: 1207890850
real    0m4.528s
user    0m4.510s
sys     0m0.006s

Adjust arguments when the run is too short for profiling. For example, ./perf-cpu-burn 38 50 increases CPU time further.

Measure a workload with perf stat

perf stat wraps a command and prints aggregate counters when the command exits.

perf stat ./perf-cpu-burn

Sample output:

Performance counter stats for './perf-cpu-burn':

          4,510.218 msec task-clock        #    0.996 CPUs utilized
                  3      context-switches
                  0      cpu-migrations
                 61      page-faults
       <not supported>      cycles
       <not supported>      instructions

       4.528400321 seconds time elapsed

       4.510000000 seconds user
       0.006000000 seconds sys

Exact counter values vary between hosts, but task-clock, elapsed time, and user or system time should stay internally consistent.

Common counters:

Counter patterns are investigation signals, not automatic diagnoses. Compare the same workload across repeated runs and confirm the hypothesis with a function profile, call graph, tracepoint, or memory recording.

Request specific events when your CPU and permissions allow them:

perf stat -e cycles,instructions,cache-references,cache-misses ./perf-cpu-burn

On the tested VM, use software counters instead because hardware PMU events were unavailable:

perf stat -e task-clock,context-switches,page-faults ./perf-cpu-burn

If an event is unavailable, perf reports an error or <not supported> for that counter. That does not always mean perf is broken—it may mean the host cannot expose that PMU event.

When both instructions and cycles are available, compare instructions per cycle (IPC) across repeated runs of the same workload. Rising cache-miss ratios alongside low IPC often point to memory behavior worth investigating. Elapsed wall time alone does not identify the hotspot function.

Repeat measurements for repeatability. The -r option runs the command multiple times and reports average and standard deviation:

perf stat -r 5 -e cycles,instructions,branches,branch-misses,cache-misses -- ./perf-cpu-burn

On hosts without hardware PMU support, substitute software events such as task-clock and page-faults.

Repeat each measurement several times on a quiet system. Background services, power management, and CPU frequency scaling can shift counter values between runs.

Find CPU hotspots with perf record and report

perf record captures samples into perf.data (by default in the current directory). perf report ranks symbols by sample overhead.

On the tested VM, hardware PMU events were unavailable, so the recordings below use the software cpu-clock event at 99 Hz instead of the default cycles event. On bare metal where cycles is supported, you can record with -e cycles or omit -e to accept the default hardware event.

Record the sample workload:

perf record -e cpu-clock -F 99 -o perf-cpu-burn.data -- ./perf-cpu-burn

Sample output:

done: 1207890850
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.040 MB perf-cpu-burn.data (430 samples) ]

Open the interactive report:

perf report -i perf-cpu-burn.data

For scripts, logs, or copy-paste review, use the text report. The --no-children option reports self overhead rather than accumulating descendant overhead into parent functions. Remove it when you want to inspect inherited call-chain cost; children overhead is enabled by default in modern perf report.

perf report -i perf-cpu-burn.data --stdio --no-children

Add --show-nr-samples when you need to see whether a profile is based on a handful of samples or a statistically useful capture:

perf report -i perf-cpu-burn.data --stdio --no-children --show-nr-samples

Sample output:

# Overhead  Samples  Command        Shared Object         Symbol
# ........  .......  .............  ....................  .....................................
#
    95.35%       410  perf-cpu-burn  perf-cpu-burn         [.] fib
     3.72%        16  perf-cpu-burn  [kernel.kallsyms]     [k] irqentry_exit_to_user_mode
     0.47%         2  perf-cpu-burn  [kernel.kallsyms]     [k] handle_softirqs
     0.47%         2  perf-cpu-burn  [kernel.kallsyms]     [k] _raw_spin_unlock_irqrestore

Key columns:

With a multi-second workload, application symbols such as fib dominate the profile. Short runs of a tiny program still spend noticeable time in the dynamic linker and kernel entry path.

Profile a running process

To attach to a running process or isolate one worker thread, use the bounded production workflow in Practical perf workflows for real problems.

Record system-wide activity

Capture all CPUs and processes:

sudo perf record -e cpu-clock -F 99 -a -o perf-system.data -- sleep 30

Read system-wide recordings with the same privilege context used to create them:

sudo perf report -i perf-system.data

System-wide recordings produce larger perf.data files and include unrelated processes. Use them when the bottleneck might move between services or when you do not know which PID to choose.

Record and interpret call graphs

Add call-graph information when you need parent and child relationships, not just flat symbol ranks.

perf record -e cpu-clock -F 99 --call-graph dwarf -o perf-cg.data -- ./perf-cpu-burn

perf report -i perf-cg.data --stdio --no-children

Sample output:

74.19%  perf-cpu-burn  perf-cpu-burn         [.] fib
            |
            ---fib
               |
               |--71.88%--fib
               |          |
               |          |--54.84%--main
               |          |          __libc_start_call_main
               |          |          __libc_start_main@@GLIBC_2.34
               |          |          _start

cpu-clock provides a software sampling fallback when a VM does not expose hardware PMU events. On bare metal where cycles is supported, you can record -e cycles or use the default event.

Common unwind methods:

Frame-pointer unwinding becomes unreliable when binaries omit frame pointers. DWARF unwinding records user-space stack data and has higher overhead. No single mode is best for every workload. Try fp on frame-pointer builds, dwarf when stacks look truncated, and lbr only when perf lists it for your CPU.

Expand parent and child chains in the interactive perf report UI to separate self overhead from inherited overhead. Broken stacks often mean missing frame pointers, stripped binaries, or incompatible unwind settings.

Resolve symbols with debuginfod

Stripped binaries and missing debuginfo produce rows like [unknown] or raw addresses in reports. debuginfod can fetch debugging data on demand when a trusted server has the matching build ID.

Check whether a URL is already configured:

echo "$DEBUGINFOD_URLS"

An empty line means no server is active in the current shell. On many distributions, /etc/profile.d/debuginfod.sh reads URL files from /etc/debuginfod/ at login. The DEBUGINFOD_URLS variable accepts a space-separated list of trusted server URLs.

If no URL file is present, set one explicitly—for example the federated public service:

export DEBUGINFOD_URLS=https://debuginfod.elfutils.org/

A public federating server may not contain debugging data for every Rocky Linux, RHEL, Ubuntu, proprietary, container, or locally built binary. Use your distribution's configured service, an internal debuginfod server, or matching local debuginfo packages when the public service has no result. Successful resolution depends on some connected server actually indexing that build ID.

The locally compiled perf-cpu-burn sample gets source and line information from the -g compile option. The debuginfod-find example below tests whether debug information for a distribution-provided system binary can be retrieved by build ID—it does not automatically add debugging information to your locally built program.

Look up debuginfo for a system binary:

build_id=$(readelf -n /usr/bin/sleep | awk '/Build ID/ {print $3; exit}')
DEBUGINFOD_URLS=https://debuginfod.elfutils.org/ debuginfod-find debuginfo "$build_id"

When the server has the requested file, debuginfod-find prints its path under the local client cache, normally ~/.cache/debuginfod_client/. Re-run perf report or perf annotate after symbols download.

Before resolution, a hot row might appear as:

[unknown]

After debuginfo is available, the same samples can show named functions or source paths—when the server hosts that build and unwinding succeeds.

[unknown] can still appear when:

• The binary was built without debug information and no server hosts it
• The code is JIT-generated
• The executable was replaced or deleted after recording
• Stack unwinding failed
• Kernel symbols are restricted
• perf_event_paranoid or permissions block full kernel visibility

Privacy note: debuginfod requests reveal build IDs of the files you ask for to the configured server. Use only trusted URLs, especially on production hosts.

Inspect source and assembly with perf annotate

perf annotate maps samples to instructions inside a function. Quality depends on matching binaries, debuginfo, compiler optimizations, available source files, and successful debuginfod retrieval.

After recording with call-graph data:

perf annotate -i perf-cg.data --stdio fib

Sample output:

Percent | Source code & Disassembly of perf-cpu-burn for cpu-clock (410 samples)
------------------------------------------------------------------
         :
         :
    91.30 :   4012a4:        callq   fib
     4.35 :   4012a9:        addq    %rax, %r12

When source lines are missing, perf annotate still shows disassembly percentages. That is enough to see which instructions attract samples inside a hot function.

Use perf top, tracepoints, and BPF-related features

Real-time profiling

On the tested VM, use the software cpu-clock event for live profiling as well:

sudo perf top -e cpu-clock -F 99

For one process:

sudo perf top -e cpu-clock -F 99 -p <PID>

On hosts where hardware cycles is supported, you can omit -e cpu-clock and use the default event.

perf top refreshes live function overhead inside the selected scope. It is useful after you have already identified the hot PID with top or ps.

System-call tracing

sudo perf trace /bin/true

Sample output:

0.245 ( 3.129 ms): true/33633 brk()                                         = 0x55a2e7672000
     8.481 ( 1.776 ms): true/33633 mmap(len: 8192, prot: READ|WRITE, ...)       = 0x7f6b03ad6000
    10.698 ( 0.095 ms): true/33633 openat(dfd: CWD, filename: "/etc/ld.so.cache") = 3

Discover events

perf list tracepoint | grep sched:sched_switch

Sample output:

sched:sched_switch                                 [Tracepoint event]

Record one tracepoint system-wide:

sudo perf record -e sched:sched_switch -a -o perf-sched.data -- sleep 10

Newer perf builds also expose kernel tracepoints, dynamic kprobes, uprobes, and BPF-related features through perf list and perf record -e. Those paths are powerful but belong in dedicated bpftrace, BCC, and advanced perf probe articles—not in this baseline workflow.

Practical perf workflows for real problems

The sections above teach individual commands. The workflows below answer a common operational question: my server or application is slow—where do I start with perf?

A. Profile a high-CPU process

When you do not know which process is hot, rank candidates with the ps command:

ps -eo pid,comm,%cpu --sort=-%cpu | head

top serves the same purpose interactively. After you choose a PID, inspect it live:

pidof my-application
sudo perf top -e cpu-clock -F 99 -p <PID>

On production systems, start with a short duration, moderate sampling frequency, and an explicit output file. A 30-second recording at 99 Hz is usually easier to control than an unlimited default session. Call-graph recording, especially DWARF stacks, increases CPU, memory, and disk overhead; validate the impact on a staging or low-risk host first. DWARF mode stores a user-space stack dump with every sample, using an 8192-byte default stack-dump size unless you change it.

Capture a bounded profile during the slow period:

sudo perf record -e cpu-clock -F 99 --call-graph dwarf -p <PID> -o app-high-cpu.data -- sleep 30

Summarize by function:

sudo perf report -i app-high-cpu.data --stdio --no-children

Check recording size and metadata before sharing or archiving:

ls -lh app-high-cpu.data
sudo perf report -i app-high-cpu.data --header-only

High self overhead means the function itself is consuming samples. High inherited or children overhead means expensive work occurs deeper in its call chain. Repeat the capture during the actual slow period, not only when the system looks idle.

When one worker thread dominates, list threads for that PID:

ps -L -p <PID> -o pid,tid,pcpu,comm

Profile a specific thread:

sudo perf record -e cpu-clock -F 99 --call-graph dwarf -t <TID> -o hot-thread.data -- sleep 30

That pattern is especially useful for Java worker pools, web-server threads, database workers, and multithreaded native applications.

B. Compare performance before and after a change

Use perf stat -r when you need repeatable counter averages:

perf stat -r 5 -- ./perf-cpu-burn

For function-level comparison, record before and after with the same event and frequency:

perf record -e cpu-clock -F 99 -o before.data -- ./perf-cpu-burn

After the code or configuration change:

perf record -e cpu-clock -F 99 -o after.data -- ./perf-cpu-burn

Compare profiles:

perf diff before.data after.data

perf diff matches event, object, and symbol information across recordings. A lower percentage for one function does not automatically mean that function became faster; another part of the program may simply consume a larger share. Compare elapsed time, counter totals, workload size, and profile percentages together.

C. Investigate an intermittent system-wide CPU spike

Capture all CPUs for a short window:

sudo perf record -e cpu-clock -F 99 -a --call-graph dwarf -o system-spike.data -- sleep 30

Summarize by process and function:

sudo perf report -i system-spike.data --stdio --sort comm,dso,symbol

When only specific CPUs appear busy, restrict sampling:

sudo perf record -e cpu-clock -F 99 -a -C 2-3 -o busy-cpus.data -- sleep 30

perf record -C restricts sampling to selected CPUs, while system-wide mode captures activity from all processes running on those CPUs. That is practical for load spikes that move between services, interrupt-heavy CPUs, or unknown processes consuming CPU during a short capture window.

D. Choose the next tool when CPU hotspots are not enough

perf mem, perf lock, and perf c2c are distinct facilities. They require suitable hardware, kernel support, tracepoints, and permissions, so treat them as next-step tools rather than baseline requirements for every host.

Save, transfer, and compare perf data

The default recording file is perf.data. Name outputs explicitly when you run multiple experiments:

perf record -e cpu-clock -F 99 -o app-hotspot.data -- ./perf-cpu-burn

Inspect a saved file:

perf report -i app-hotspot.data

Check file size and capture metadata before sharing or archiving:

ls -lh app-hotspot.data
perf report -i app-hotspot.data --header-only

Export sample lines for scripting:

perf script -i app-hotspot.data | head

Sample output:

perf-cpu-burn   33431  5179.798755: cpu-clock:      7ff078b0cc6b fib+0x24 (/tmp/perf-cpu-burn)
perf-cpu-burn   33431  5179.799128: cpu-clock: ffffffff9a192c7a finish_task_switch.isra.0+0x9a ([kernel.kallsyms])

List build IDs referenced by a recording:

perf buildid-list -i app-hotspot.data

Sample output:

0f676d64cf26845532e461effe9b686000eff444 /tmp/perf-cpu-burn
5d09d0419d3fe4a68200b434ebb1f8dc0f9db206 /usr/lib64/ld-linux-x86-64.so.2
d7cb3dcb41fb386b8c087ecab77968f6adafacea [kernel.kallsyms]

The header from --header-only records hostname, kernel version, perf version, event definitions, and sample timing—useful when you analyze recordings offline. Use perf diff when you need to compare two saved recordings from before-and-after experiments.

When moving data to another machine:

• Keep the same binaries or matching build IDs
• Install the same perf major version when possible
• Use perf archive for controlled offline bundles when your workflow requires it
• Compare only runs with similar workload intensity and duration

Flame-graph generation is a useful next step but belongs in a separate visualization article.

Troubleshoot perf and debuginfod problems

Useful diagnostic commands:

cat /proc/sys/kernel/perf_event_paranoid

cat /proc/sys/kernel/kptr_restrict

perf list

perf report -i perf.data --header-only

Discuss security implications with your team before lowering perf_event_paranoid or kptr_restrict permanently. Temporary sudo perf record is usually safer on shared servers than opening kernel profiling to every local user.

References

• perf wiki — main page
• Linux kernel trace documentation
• perf-stat(1)
• perf-record(1)
• perf-report(1)
• perf-annotate(1)
• elfutils debuginfod documentation
• Red Hat Enterprise Linux 10.2 release notes
• Red Hat Enterprise Linux 10 — Monitoring and managing system status and performance
• Ubuntu Manpage — debuginfod-client-config
• Debian Package: linux-perf

Summary

Install the distribution perf package, confirm kernel support with perf version, uname -r, and perf list, then profile workloads in order: perf stat for counters, perf record and perf report for hot functions, and --call-graph when you need stacks. On VMs without hardware PMU counters, use -e cpu-clock -F 99 for perf record and perf top. Configure DEBUGINFOD_URLS when reports show [unknown] and you trust a debuginfod server. Use the practical workflows section for high-CPU processes, before/after comparisons, system-wide spikes, and choosing the next specialized perf tool. Name output files explicitly, check recording size with ls -lh, and treat perf_event_paranoid changes as a security decision—not a default tuning step.