7 tools to detect Memory Leaks with Examples


Written by - Deepak Prasad

In this tutorial I will share different methods and tools to detect and find memory leaks with different processes in Linux. As a developer we often face scenarios when proess such as httpd apache, java starts consuming high amount of memory leading to OOM (Out Of memory) situations.

So it is always healthy to keep monitoring the memory usage of critical process. I work for an application which is memory intensive so it is my job to make sure other processes are not eating up the memory unnecessarily. In this process I use different tools in real time environments to detect memory leaks and then report it to the responsible developers.

 

What is Memory Leak?

  • Memory is allocated on demand—using malloc() or one of its variants—and memory is freed when it’s no longer needed.
  • A memory leak occurs when memory is allocated but not freed when it is no longer needed.
  • Leaks can obviously be caused by a malloc() without a corresponding free(), but leaks can also be inadvertently caused if a pointer to dynamically allocated memory is deleted, lost, or overwritten.
  • Buffer overruns—caused by writing past the end of a block of allocated memory—frequently corrupt memory.
  • Memory leakage is by no means unique to embedded systems, but it becomes an issue partly because targets don't have much memory in the first place and partly because they often run for long periods of time without rebooting, allowing the leaks to become a large puddle.
  • Regardless of the root cause, memory management errors can have unexpected, even devastating effects on application and system behavior.
  • With dwindling available memory, processes and entire systems can grind to a halt, while corrupted memory often leads to spurious crashes.

 

Sample program to add memory leak

Here is a simple C program that leaks memory

#include <stdio.h>
#include <stdlib.h>

int main() {
    for (int i = 0; i < 10; i++) {
        int *leak = (int *) malloc(sizeof(int) * 10);
        printf("Memory allocated at iteration %d\n", i);
    }
    return 0;
}

This program allocates memory for an integer array of size 10 but does not free it, causing a memory leak. Compile the program with gcc -o memleak memleak.c.

Before you go ahead I would recommend you to also read about Linux memory management so that you are familiar with the different terminologies used in Linux kernel in terms of memory.

 

1. Valgrind

  • Valgrind is an Intel x86-specific tool that emulates an x86-class CPU to watch all memory accesses directly and analyze data flow
  • One advantage is that you don't have to recompile the programs and libraries that you want to check, although it works better if they have been compiled with the -g option so that they include debug symbol tables.
  • It works by running the program in an emulated environment and trapping execution at various points.
  • This leads to the big downside of Valgrind, which is that the program runs at a fraction of normal speed, which makes it less useful in testing anything with real-time constraints.

Valgrind can detect problems such as:

  • Use of uninitialized memory
  • Reading and writing memory after it has been freed
  • Reading and writing from memory past the allocated size
  • Reading and writing inappropriate areas on the stack
  • Memory leaks
  • Passing of uninitialized and/or unaddressable memory
  • Mismatched use of malloc/new/new() versus free/delete/delete()

Installation:

  • On Ubuntu/Debian: sudo apt-get install valgrind
  • On Fedora: sudo dnf install valgrind
  • On CentOS/RHEL: sudo yum install valgrind
  • On openSUSE: sudo zypper install valgrind
  • On Arch/Manjaro: sudo pacman -S valgrind

Syntax:

valgrind --leak-check=full ./my_program

Run the compiled program with Valgrind's Memcheck tool:

# valgrind --leak-check=full ./memleak
==6420== Memcheck, a memory error detector
==6420== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==6420== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==6420== Command: ./memleak
==6420== 
Memory allocated at iteration 0
Memory allocated at iteration 1
Memory allocated at iteration 2
Memory allocated at iteration 3
Memory allocated at iteration 4
Memory allocated at iteration 5
Memory allocated at iteration 6
Memory allocated at iteration 7
Memory allocated at iteration 8
Memory allocated at iteration 9
==6420== 
==6420== HEAP SUMMARY:
==6420==     in use at exit: 400 bytes in 10 blocks
==6420==   total heap usage: 11 allocs, 1 frees, 1,424 bytes allocated
==6420== 
==6420== 400 bytes in 10 blocks are definitely lost in loss record 1 of 1
==6420==    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==6420==    by 0x109187: main (in /root/memleak)
==6420== 
==6420== LEAK SUMMARY:
==6420==    definitely lost: 400 bytes in 10 blocks
==6420==    indirectly lost: 0 bytes in 0 blocks
==6420==      possibly lost: 0 bytes in 0 blocks
==6420==    still reachable: 0 bytes in 0 blocks
==6420==         suppressed: 0 bytes in 0 blocks
==6420== 
==6420== For lists of detected and suppressed errors, rerun with: -s
==6420== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

As expected, the memleak program allocates memory 10 times in the loop.

HEAP SUMMARY: This section provides a summary of the memory usage at the exit point of the program. It shows that there are 400 bytes of memory still in use (not freed) and that there were a total of 11 memory allocations and 1 free operation.

The memory leak details: Valgrind reports that 400 bytes in 10 blocks are definitely lost, which corresponds to the memory leak in the program. The output also provides information about the location in the source code where the memory was allocated:

The memory leak occurred at 0x109187, which is the main function in the memleak program.

The memory allocation occurred at 0x4848899, which is the malloc function in the Valgrind memory-checking library.

LEAK SUMMARY: This section provides a summary of the detected memory leaks. It shows that there are 400 bytes in 10 blocks that are definitely lost and no other memory issues detected.

ERROR SUMMARY: Valgrind reports a total of 1 error from 1 context. In this case, the error refers to the memory leak that was detected.

 

2. GDB (GNU Debugger)

GDB is a widely-used, open-source debugger for Linux and other platforms that supports multiple languages, including C, C++, Objective-C, Fortran, and others. It can help you find memory leaks by setting breakpoints, inspecting memory allocations, and analyzing core dumps. GDB is open-source software, distributed under the GPLv3 license.

Features:

  • Supports a wide range of programming languages.
  • Allows setting breakpoints, stepping through code, and examining variables.
  • Can debug programs at the source code and assembly levels.
  • Provides scripting support using Python or Guile.

Installation:

  • On Ubuntu/Debian: sudo apt-get install gdb
  • On Fedora: sudo dnf install gdb
  • On CentOS/RHEL: sudo yum install gdb
  • On openSUSE: sudo zypper install gdb
  • On Arch/Manjaro: sudo pacman -S gdb

Syntax:

gdb ./my_program

Example:

Compile the program with debugging symbols using the -g flag:

gcc -g -o memleak memleak.c

Run the program using GDB:

gdb ./memleak

Set a breakpoint at the line where the memory allocation occurs, which is the malloc call. In this example, let's assume the malloc call is at line 8. To set the breakpoint, use the break command:

(gdb) break 6
Breakpoint 1 at 0x117e: file memleak.c, line 6.

Run the program with the run command:

(gdb) run
Starting program: /root/memleak 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Breakpoint 1, main () at memleak.c:6
6 int *leak = (int *) malloc(sizeof(int) * 10);

The program will stop at the breakpoint (the malloc call) in the first iteration of the loop. Now, examine the memory allocation by printing the value of the leak pointer:

(gdb) print leak
$1 = (int *) 0x0

To continue the execution, use the continue command:

(gdb) continue
Continuing.
Memory allocated at iteration 0

Breakpoint 1, main () at memleak.c:6
6 int *leak = (int *) malloc(sizeof(int) * 10)

Repeat steps print leak and continue for each iteration of the loop to examine the memory allocations.

(gdb) print leak
$2 = (int *) 0x5555555592a0
(gdb) continue
Continuing.
Memory allocated at iteration 1

Breakpoint 1, main () at memleak.c:6
6 int *leak = (int *) malloc(sizeof(int) * 10);
(gdb) print leak
$3 = (int *) 0x5555555596e0
(gdb) continue
Continuing.
Memory allocated at iteration 2

Breakpoint 1, main () at memleak.c:6
6 int *leak = (int *) malloc(sizeof(int) * 10);
(gdb) print leak
$4 = (int *) 0x555555559710
(gdb) continue
Continuing.
Memory allocated at iteration 3

Breakpoint 1, main () at memleak.c:6
6 int *leak = (int *) malloc(sizeof(int) * 10);
(gdb) print leak
$5 = (int *) 0x555555559740
(gdb) continue
Continuing.
Memory allocated at iteration 4

Breakpoint 1, main () at memleak.c:6
6 int *leak = (int *) malloc(sizeof(int) * 10);

Once you have completed all iterations of the loop and the program has finished executing, you can observe that the allocated memory is never freed, which indicates a memory leak. As mentioned earlier, GDB doesn't automatically detect memory leaks, but this manual examination process can help you identify potential issues.

 

3. LeakSanitizer (AddressSanitizer)

LeakSanitizer is a memory leak detector that is integrated into AddressSanitizer, a runtime memory error detection tool. AddressSanitizer (ASan) is designed to find out-of-bounds access, use-after-free, and other memory-related issues in C/C++ programs.

Features:

  • Fast execution, with only 2x slowdown in most cases
  • Low memory overhead, typically 3x
  • Can detect various memory errors, such as leaks, use-after-free, and buffer overflows

To use AddressSanitizer with LeakSanitizer, compile the program with AddressSanitizer flags:

gcc -fsanitize=address -fno-omit-frame-pointer -g -o memleak memleak.c

Run the program:

# ./memleak 
Memory allocated at iteration 0
Memory allocated at iteration 1
Memory allocated at iteration 2
Memory allocated at iteration 3
Memory allocated at iteration 4
Memory allocated at iteration 5
Memory allocated at iteration 6
Memory allocated at iteration 7
Memory allocated at iteration 8
Memory allocated at iteration 9

=================================================================
==71412==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 400 byte(s) in 10 object(s) allocated from:
    #0 0x7f0860216867 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
    #1 0x55cae8e4e207 in main /root/memleak.c:6
    #2 0x7f085ff63d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58

SUMMARY: AddressSanitizer: 400 byte(s) leaked in 10 allocation(s).

Here we can see that LeakSanitizer successfully detected memory leaks in our memleak program. The program allocated memory 10 times in the loop but never freed it, resulting in a direct leak of 400 bytes.

The output provides detailed information about the memory leaks, including the size of the leak, the number of objects involved, and the call stack that led to the allocation. In this case, the allocation happened in the main function at line 6 of memleak.c.

 

4. mtrace

mtrace is a memory leak detection tool that is part of the GNU C Library. It traces memory allocations and deallocations performed by a program and reports any detected memory leaks.

Features:

  • Can detect memory leaks in C programs
  • Minimal overhead during execution

To use mtrace, the code should include the mcheck.h header and call the mtrace() function at the beginning of the main() function and muntrace() at the end of the main() function.

#include <stdio.h>
#include <stdlib.h>
#include <mcheck.h>

int main() {
    mtrace(); // Start tracing memory allocations

    for (int i = 0; i < 10; i++) {
        int *leak = (int *) malloc(sizeof(int) * 10);
        printf("Memory allocated at iteration %d\n", i);
    }

    muntrace(); // Stop tracing memory allocations
    return 0;
}

After modifying the code, compile the program and run it again:

gcc -g -o memleak memleak.c

Set the MALLOC_TRACE environment variable to specify the output file for mtrace:

export MALLOC_TRACE=memleak_output

Run the program:

# ./memleak
Memory allocated at iteration 0
Memory allocated at iteration 1
Memory allocated at iteration 2
Memory allocated at iteration 3
Memory allocated at iteration 4
Memory allocated at iteration 5
Memory allocated at iteration 6
Memory allocated at iteration 7
Memory allocated at iteration 8
Memory allocated at iteration 9

Memory not freed:
-----------------
Address     Size        Caller
0x12345678  40 bytes    main (memleak.c:12)
0x23456789  80 bytes    main (memleak.c:16)
...

In the output, it lists the memory allocations that were not freed, along with their addresses, sizes, and the corresponding caller information (function name and file:line). This helps you identify the specific locations in your code where the memory leaks occurred.

 

5. Memleax

One of the drawbacks of Valgrind is that you cannot check memory leak of an existing process which is where memleax comes for the rescue. I have had instances where the memory leak was very sporadic for amsHelper process so at times when I see this process reserving memory I wanted to debug that specific PID instead of creating a new one for analysis.

memleax  debugs memory leak of a running process by attaching it. It hooks the target process's invocation of memory allocation and free, and reports the memory blocks which live long enough as memory leak, in real time. The default expire threshold is 10 seconds, however you should always set it by -e option according to your scenarios.

You can download memleax from the official github repository
Next memleax expects few dependencies which you must install before installing the memleax rpm

# rpm -Uvh /tmp/memleax-1.1.1-1.el7.centos.x86_64.rpm
error: Failed dependencies:
        libdwarf.so.0()(64bit) is needed by memleax-1.1.1-1.el7.centos.x86_64
        libunwind-x86_64.so.8()(64bit) is needed by memleax-1.1.1-1.el7.centos.x86_64
        libunwind.so.8()(64bit) is needed by memleax-1.1.1-1.el7.centos.x86_64

So I have manually copied these rpms from my official repository as this is a private network I could not use yum or dnf

# rpm -Uvh /tmp/libdwarf-20130207-4.el7.x86_64.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:libdwarf-20130207-4.el7          ################################# [100%]

# rpm -Uvh /tmp/libunwind-1.2-2.el7.x86_64.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:libunwind-2:1.2-2.el7            ################################# [100%]

Now since I have installed both the dependencies, I will go ahead and install memleax rpm:

# rpm -Uvh /tmp/memleax-1.1.1-1.el7.centos.x86_64.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:memleax-1.1.1-1.el7.centos       ################################# [100%]

Next you need the PID of the process which you wish to monitor. You can get the PID of your process from ps -ef output

root      2102     1  0 12:29 ?        00:00:01 /sbin/amsHelper -f
root     45256     1  0 13:13 ?        00:00:00 amsHelper
root     49372 44811  0 13:23 pts/0    00:00:00 grep amsH

Now here we wish to check the memory leak of 45256 PID

# memleax 45256
Warning: no debug-line found for /usr/sbin/amsHelper
== Begin monitoring process 45256...
CallStack[1]: memory expires with 688 bytes, backtrace:
    0x00007f0bc87010d0  libc-2.17.so  calloc()+0
    0x00000000004079d3  amsHelper
    0x0000000000409249  amsHelper
    0x0000000000407077  amsHelper
    0x00000000004a7a60  amsHelper
    0x00000000004a8c4c  amsHelper
    0x00000000004afd90  amsHelper
    0x00000000004ac97a  amsHelper  table_helper_handler()+2842
    0x00000000004afd90  amsHelper
    0x00000000004bae09  amsHelper
    0x00000000004bb707  amsHelper
    0x00000000004bb880  amsHelper
    0x00000000004bbca2  amsHelper
    0x00000000004e7eb1  amsHelper
    0x00000000004e8a3e  amsHelper
    0x00000000004e98a9  amsHelper
    0x00000000004e98fb  amsHelper
    0x00000000004051b4  amsHelper
    0x00007f0bc869d555  libc-2.17.so  __libc_start_main()+245
    0x00000000004053e2  amsHelper
CallStack[1]: memory expires with 688 bytes, 2 times again
CallStack[1]: memory expires with 688 bytes, 3 times again
CallStack[1]: memory expires with 688 bytes, 4 times again
CallStack[1]: memory expires with 688 bytes, 5 times again
CallStack[2]: memory expires with 15 bytes, backtrace:
    0x00007f0bc87006b0  libc-2.17.so  malloc()+0
    0x00007f0bc8707afa  libc-2.17.so  __GI___strdup()+26
    0x00007f0bc8731141  libc-2.17.so  tzset_internal()+161
    0x00007f0bc8731b03  libc-2.17.so  __tz_convert()+99

You may get output similar to above in case of a memory leak in the application process. Press Ctrl+C to stop monitoring

 

6. Collecting core dump

It helps for the developer at times we can share the core dump of the process which is leaking memory. In Red Hat/CentOS you can collect core dump using abrt and abrt-addon-ccpp
Before you start make sure the system is set up to generate application cores by removing the core limits:

# ulimit -c unlimited

Next install these rpms in your environment

# yum install abrt abrt-addon-ccpp abrt-tui

Ensure the ccpp hooks are installed:

# abrt-install-ccpp-hook install
# abrt-install-ccpp-hook is-installed; echo $?;

Ensure that this service is up and the ccpp hook to capture core dumps is enabled:

# systemctl enable abrtd.service --now
# systemctl enable abrt-ccpp.service --now

Enable the hooks

# abrt-auto-reporting enabled

To get a list of crashes on the command line, issue the following command:

# abrt-cli list

But since there are no crash the output would be empty. Next get the PID for which you wish to collect core dump, here for example I will collect for PID 45256

root      2102     1  0 12:29 ?        00:00:01 /sbin/amsHelper -f
root     45256     1  0 13:13 ?        00:00:00 amsHelper
root     49372 44811  0 13:23 pts/0    00:00:00 grep amsH

Next we have to send SIGABRT i.e. -6 kill signal to this PID to generate the core dump

# kill -6 45256

Next you can check the list of available dumps, now you can see a new entry for this PID. This dump will contain all the information required to analyse the leak for this process

# abrt-cli list
id 2b9bb9702d83e344bc940b813b43262ede9d9521
reason:         amsHelper killed by SIGABRT
time:           Thu 13 Aug 2020 02:29:27 PM +0630
cmdline:        amsHelper
package:        hp-ams-2.10.0-861.6.rhel7
uid:            0 (root)
Directory:      /var/spool/abrt/ccpp-2020-08-13-14:29:27-45256

 

7. How to identify memory leak using default Linux tools

We discussed about third party tools which can be used to detect memory leak with more information in the code which can help the developer analyse and fix the bug. But if our requirement is just to look out for process which is reserving memory for no reason then we will have to rely on system tools such as sar, vmstat, pmap, meminfo etc

So let's learn about using these tools to identify a possible memory leak scenario. Before you start you must be familiar with below areas

  1. How to check the actual memory consumed by individual process
  2. How much memory reservation is normal for your application process

If you have answers to above questions then it will be easier to analyse the problem.

For example in my case I know the memory usage of amsHelper should not be more than few MB but in case of memory leak the memory reservation goes way higher than few MB. if you have read my earlier article where I explained different tools to check actual memory usage, you would know that Pss gives us an actual idea of the memory consumed by the process.

 

<strong>pmap</strong> would give you more detailed output of memory consumed by individual address segments and libraries of the process as you can see below

# pmap -X $(pgrep amsHelper -f)
15046:   /sbin/amsHelper -f
         Address Perm   Offset Device  Inode   Size   Rss   Pss Referenced Anonymous Swap Locked Mapping
        00400000 r-xp 00000000  fd:02   7558   1636  1152  1152       1152         0    0      0 amsHelper
        00799000 r--p 00199000  fd:02   7558      4     4     4          4         4    0      0 amsHelper
        0079a000 rw-p 0019a000  fd:02   7558     52    48    48         48        20    0      0 amsHelper
        007a7000 rw-p 00000000  00:00      0    356    48    48         48        48    0      0
        01962000 rw-p 00000000  00:00      0   9716  9716  9716       9716      9716    0      0 [heap]
    7fd75048b000 r-xp 00000000  fd:02   3406    524   320    44        320         0    0      0 libfreeblpriv3.so
    7fd75050e000 ---p 00083000  fd:02   3406   2048     0     0          0         0    0      0 libfreeblpriv3.so
    7fd75070e000 r--p 00083000  fd:02   3406      8     8     8          8         8    0      0 libfreeblpriv3.so
    7fd750710000 rw-p 00085000  fd:02   3406      4     4     4          4         4    0      0 libfreeblpriv3.so
    7fd750711000 rw-p 00000000  00:00      0     16    16    16         16        16    0      0

<output trimmed>

    7fd75ba5f000 rw-p 00022000  fd:02   4011      4     4     4          4         4    0      0 ld-2.17.so
    7fd75ba60000 rw-p 00000000  00:00      0      4     4     4          4         4    0      0
    7ffdeb75d000 rw-p 00000000  00:00      0    132    32    32         32        32    0      0 [stack]
    7ffdeb79a000 r-xp 00000000  00:00      0      8     4     0          4         0    0      0 [vdso]
ffffffffff600000 r-xp 00000000  00:00      0      4     0     0          0         0    0      0 [vsyscall]
                                             ====== ===== ===== ========== ========= ==== ======
                                             196632 15896 13896      15896     10384    0      0 KB

Alternatively you can get the same information with more details using smaps of the respective process. Here I have written a small script to combine the memory and get the total but you can also remove the pipe and break down the command to get more details

# cat /proc/$(pgrep amsHelper)/smaps | grep -i pss |  awk '{Total+=$2} END {print Total/1024" MB"}'
14.4092 MB

So you can put a cron job or create a daemon to timely monitor the memory consumption of your application using these tools to figure out if they are consuming too much memory over time.

 

Conclusion

In this tutorial I shared different commands, tools and methods to detect and monitor memory leak across different types of applications such as C or C++ programs, Linux applications etc. The selection of tools would vary based on your requirement. There are many other tools such as YAMD, Electric fence, gdb core dump etc which can help you capture memory leak.

Lastly I hope the steps from the article to check and monitor memory leak on Linux was helpful. So, let me know your suggestions and feedback using the comment section.

 

References

I have used below external references for this tutorial guide
Memleax - Debugging memory leak

 

Deepak Prasad

He is the founder of GoLinuxCloud and brings over a decade of expertise in Linux, Python, Go, Laravel, DevOps, Kubernetes, Git, Shell scripting, OpenShift, AWS, Networking, and Security. With extensive experience, he excels in various domains, from development to DevOps, Networking, and Security, ensuring robust and efficient solutions for diverse projects. You can reach out to him on his LinkedIn profile or join on Facebook page.

Can't find what you're searching for? Let us assist you.

Enter your query below, and we'll provide instant results tailored to your needs.

If my articles on GoLinuxCloud has helped you, kindly consider buying me a coffee as a token of appreciation.

Buy GoLinuxCloud a Coffee

For any other feedbacks or questions you can send mail to admin@golinuxcloud.com

Thank You for your support!!

2 thoughts on “7 tools to detect Memory Leaks with Examples”

Leave a Comment

X