How to PROPERLY check memory usage per process in Linux?

In Linux, checking the memory usage per process is an essential task to ensure optimal system performance. Monitoring memory usage can help identify processes that are consuming too much memory or experiencing memory leaks, which can lead to system instability and performance degradation. Fortunately, Linux provides several built-in tools to monitor and analyze memory usage per process.

There are different methods to check memory usage per process in Linux, including command-line tools, graphical user interfaces (GUIs), and third-party tools.

Command-line tools such as 'ps', 'top', and 'htop' provide detailed information about running processes and their memory usage. They are lightweight and efficient, making them an excellent choice for monitoring memory usage on a Linux system. These tools can be used in a terminal or shell session, making them accessible from remote connections.

GUI tools such as GNOME System Monitor, KDE System Monitor, and Xfce Task Manager provide a graphical interface for monitoring memory usage. They display the memory usage of running processes in a more intuitive and user-friendly way, making them an excellent choice for novice users.

Third-party tools such as Nagios, Zabbix, and Cacti provide advanced memory monitoring capabilities, including alerting, trend analysis, and historical data. These tools are often used in enterprise environments, where managing multiple servers and applications is a critical task.

In this guide, we will focus on using different CLI tools to check memory usage per process in Linux. By the end of this guide, you will have a better understanding of how to monitor and optimize memory usage on your Linux system, regardless of the method you choose.

You can also read Shell script to check top memory & cpu consuming process in Linux

 

Important Terminologies

In general you must be familiar with the Linux Memory Management but I will try to explain some of the important terminologies which defines the type of memory consumed by Linux processes.

There are several metrics available to check memory usage per process in Linux. I will begin with the two that are easiest to obtain

• VSZ (Virtual Memory Size) refers to the total amount of virtual memory that a process is using. This includes both the portion that is in RAM and the portion that has been swapped out to disk. It also includes any shared libraries that the process is using.
• RSS (Resident Set Size) refers to the portion of a process's memory that is held in RAM. This includes both the code and data segments of the process. It does not include memory that has been swapped out to disk.

In 2009, Matt Mackall began looking at the problem of accounting for shared pages in process memory measurement and added two new metrics called the Unique Set Size or Uss, and the Proportional Set Size or Pss

• Uss: This is the amount of memory that is committed to physical memory and is unique to a process; it is not shared with any other. It is the amount of memory that would be freed if the process were to terminate. USS is a useful metric for determining the actual memory usage of a process, as it doesn't include shared memory that might be double-counted when looking at memory usage across multiple processes.
• Pss: This splits the accounting of shared pages that are committed to physical memory between all the processes that have them mapped. For example, if an area of library code is 12 pages long and is shared by six processes, each will accumulate two pages in Pss. Thus, if you add the Pss numbers for all processes, you will get the actual amount of memory being used by those processes. In other words,Pss is the number we have been looking for.

 

Method-1: Using ps command

The following are some of the most commonly used options to check memory usage per process with the 'ps' command:

• 'e' option: This option shows information about all running processes on the system, regardless of the user.
• 'o' option: This option lets you specify the output format of the 'ps' command. You can use it to select the columns you want to see, including memory usage.
• 'sort' option: This option lets you sort the output of the 'ps' command based on a specific column, such as memory usage.

For example, the following command shows the memory usage of all processes on the system, sorted by memory usage in descending order:

ps -e -o pid,user,%mem,cmd --sort=-%mem

In this command, the '-e' option shows all processes, the '-o' option selects the columns we want to display (process ID, user, memory usage, and command), and the '--sort' option sorts the output by memory usage in descending order.

You can modify the command to list the top 10 memory consuming process in Linux. This command will give you a list of processes sorted by memory usage, but the memory usage will be displayed in percentages.

ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%mem | head

If you want to get the memory usage in human readable format such as MB/GB, then we can use awk to convert the metrics:

~]# ps -eo user,pid,ppid,cmd,pmem,rss --no-headers --sort=-rss | awk '{if ($2 ~ /^[0-9]+$/ && $6/1024 >= 1) {printf "PID: %s, PPID: %s, Memory consumed (RSS): %.2f MB, Command: ", $2, $3, $6/1024; for (i=4; i<=NF; i++) printf "%s ", $i; printf "\n"}}'
PID: 4661, PPID: 4593, Memory consumed (RSS): 964.90 MB, Command: /usr/local/bin/kube-apiserv 1.5 988060 
PID: 3223, PPID: 1, Memory consumed (RSS): 109.23 MB, Command: /usr/bin/containerd 0.1 111856 
PID: 4819, PPID: 4753, Memory consumed (RSS): 109.20 MB, Command: /usr/local/bin/kube-schedul 0.1 111816 
PID: 13597, PPID: 12093, Memory consumed (RSS): 97.79 MB, Command: /usr/src/app/env/bin/python 0.1 100132 
PID: 15540, PPID: 13597, Memory consumed (RSS): 93.56 MB, Command: /usr/src/app/env/bin/python 0.1 95808 
PID: 11966, PPID: 13597, Memory consumed (RSS): 93.32 MB, Command: /usr/src/app/env/bin/python 0.1 95556 
PID: 15476, PPID: 13597, Memory consumed (RSS): 93.29 MB, Command: /usr/src/app/env/bin/python 0.1 95528 
PID: 13413, PPID: 12938, Memory consumed (RSS): 92.23 MB, Command: /app/cmd/cainjector/cainjec 0.1 94448 
PID: 4956, PPID: 4787, Memory consumed (RSS): 88.25 MB, Command: /usr/local/bin/kube-control 0.1 90368 
PID: 15610, PPID: 13597, Memory consumed (RSS): 86.41 MB, Command: /usr/src/app/env/bin/python 0.1 88484 
PID: 15569, PPID: 13597, Memory consumed (RSS): 86.41 MB, Command: /usr/src/app/env/bin/python 0.1 88480 
PID: 15483, PPID: 13597, Memory consumed (RSS): 86.38 MB, Command: /usr/src/app/env/bin/python 0.1 88456 
PID: 15481, PPID: 13597, Memory consumed (RSS): 86.38 MB, Command: /usr/src/app/env/bin/python 0.1 88452 
PID: 15480, PPID: 13597, Memory consumed (RSS): 86.37 MB, Command: /usr/src/app/env/bin/python 0.1 88444 
PID: 15479, PPID: 13597, Memory consumed (RSS): 86.36 MB, Command: /usr/src/app/env/bin/python 0.1 88436 
PID: 15477, PPID: 13597, Memory consumed (RSS): 86.36 MB, Command: /usr/src/app/env/bin/python 0.1 88432 
...

Here I have added a filter to hide the processes with very low memory usage, you can adjust the condition in the awk command to a higher threshold. For example, you can hide processes with memory usage less than 1 MB by changing the condition to $6/1024 < 1. You can further adjust the threshold by changing the value in the condition $6/1024 >= 1 to your desired value. For example, to hide processes with memory usage less than 0.1 MB, use $6/1024 >= 0.1.

Let's understand the awk syntax:

• awk: This is the command to run the AWK text-processing tool. AWK is a scripting language used for manipulating data and generating reports.
• {...}: The curly braces are used to enclose the entire AWK script.
• if ($2 ~ /^[0-9]+$/ && $6/1024 >= 1): This is the if statement with two conditions. The first condition checks if the second field (PID) is a number ($2 ~ /^[0-9]+$/). The second condition checks if the sixth field (RSS) divided by 1024 (to convert it to MB) is greater than or equal to 1 ($6/1024 >= 1).
• printf "PID: %s, PPID: %s, Memory consumed (RSS): %.2f MB, Command: ", $2, $3, $6/1024;: This printf statement is used to print the formatted output. The %s and %.2f are placeholders for string and floating-point number values, respectively. The values of the second field (PID), third field (PPID), and sixth field (RSS) divided by 1024 (to convert it to MB) are passed as arguments.
• for (i=4; i<=NF; i++) printf "%s ", $i;: This for loop is used to print the command (from the fourth field to the last field). NF is a built-in AWK variable that represents the number of fields in the current record. The loop iterates through each field from the fourth field to the last field and prints the value of the field ($i) followed by a space.
• printf "\n": This printf statement is used to print a newline character after each record to separate the outputs for each process.

 

Method-2: Using top command

top is a real-time command-line utility that provides a dynamic, live view of the processes running on a system. It can be used to monitor system performance, including memory usage per process. Below is a detailed explanation of using top to check memory usage per process and different variations of the command.

To sort processes by memory usage:

top -o %MEM

This command sorts the processes by memory usage (%MEM) in descending order. You can also do this interactively while top is running by pressing 'M' (Shift + m).

To sort the process by memory usage and also show the commands used by the process:

top -o %MEM -c

This command sets the refresh interval for top to 2 seconds. By default, top updates every 3 seconds. You can change the refresh interval by specifying a different value after the -d flag.

top -o %MEM -c -d 2

To display process for specific user:

top -o %MEM -c -d 2 -u username

 

Method-3: Using pmap command

pmap is a command-line utility that reports the memory map of a process. It displays the memory regions allocated to a process, including the size, permissions, and mapping location.

To use pmap, you'll need to provide the process ID (PID) of the process you want to check memory usage for. Here's an example:

pmap -x [PID]

Replace [PID] with the actual process ID. The -x option shows the extended format, which includes the size, resident set size (RSS), shared, and private memory usage of each mapped memory region.

At the bottom of the output, you'll find a summary of the memory usage, including the total size, resident set size (RSS), shared, and private memory.

Here's an example of the pmap output:

# pmap -x 16411
Address           Kbytes     RSS   Dirty Mode  Mapping
0000564dc1374000   22732   20076       0 r-x-- ceph-osd
0000564dc2ba7000     296     296     296 r---- ceph-osd
0000564dc2bf1000      44      44      24 rw--- ceph-osd
0000564dc2bfc000  133328    2224    2224 rw---   [ anon ]
0000564dcc6ea000 2247272 2007596 2007596 rw---   [ anon ]
00007efdf6d9e000       4       0       0 -----   [ anon ]
00007efdf6d9f000    8192      20      20 rw---   [ anon ]
...
00007fffab7ab000       8       4       0 r-x--   [ anon ]
ffffffffff600000       4       0       0 r-x--   [ anon ]
---------------- ------- ------- ------- 
total kB         3002448 2047836 2015360

Please note that pmap gives you a detailed view of the memory usage for a single process. If you want to check memory usage for multiple processes, you would need to run pmap for each process separately,

 

Method-4: Manually check /proc/<PID>/

You can get memory usage information for a specific process from the /proc/[PID]/ directory. The /proc filesystem provides a wealth of information about a process, and the memory usage can be obtained from /proc/[PID]/statm and /proc/[PID]/status files.

 

/proc/[PID]/statm

This file shows memory usage in terms of memory pages. The second value in the file represents the resident set size (RSS). You can obtain this value using the following command:

awk '{print $2}' /proc/[PID]/statm

Replace [PID] with the actual process ID. To convert the value to kilobytes, multiply it by the page size (usually 4 KB on x86 and x86_64 systems):

pagesize=$(getconf PAGESIZE)
rss_pages=$(awk '{print $2}' /proc/[PID]/statm)
rss_kb=$((rss_pages * pagesize / 1024))
echo "RSS: $rss_kb KB"

 

/proc/[PID]/status

This file contains more human-readable information about the process. To get the resident set size (RSS) from this file, you can use the following command:

grep 'VmRSS' /proc/[PID]/status

Replace [PID] with the actual process ID. This command will display the RSS value in kilobytes, like this:

# grep -i vmrss /proc/16411/status 
VmRSS: 2018216 kB

 

/proc/[PID]/smaps

The /proc/[PID]/smaps file provides information about each memory mapping associated with the process, including the memory usage of each mapping.

For example, to get the resident set size (RSS) from the /proc/[PID]/smaps file, you can use the following command:

grep -e '^Rss' /proc/[PID]/smaps | awk '{sum += $2} END {print sum, "kB"}'

Replace [PID] with the actual process ID. This command will display the RSS value in kilobytes. The grep command filters the lines that start with 'Rss', and the awk command calculates the sum of these RSS values for all memory mappings.

For Example:

grep -e '^Rss' /proc/16411/smaps | awk '{sum += $2} END {print sum, "kB"}'
2047252 kB

To get the same value in MB we divide the total by 1024:

grep -e '^Rss' /proc/16411/smaps | awk '{sum += $2} END {print sum / 1024, "MB"}'
2034.05 MB

 

To get the USS value from the /proc/[PID]/smaps file. we need to sum the Private_Clean and Private_Dirty memory values for each memory mapping in the smaps file. Here's a command to calculate the USS value in kilobytes:

grep -e '^Private' /proc/[PID]/smaps | awk '{sum += $2} END {print sum, "kB"}'

Replace [PID] with the actual process ID. The grep command filters the lines that start with 'Private', and the awk command calculates the sum of these private memory values (both clean and dirty) for all memory mappings.

To display the USS value in megabytes, you can modify the command as follows:

grep -e '^Private' /proc/[PID]/smaps | awk '{sum += $2} END {print sum / 1024, "MB"}'

Again, replace [PID] with the actual process ID. The awk command will sum the private memory values and convert the result to megabytes by dividing the sum by 1024.

For Example:

# grep -e '^Private' /proc/16411/smaps | awk '{sum += $2} END {print sum / 1024, "MB"}'
1826.15 MB

 

You can obtain the PSS value from the /proc/[PID]/smaps file. The PSS value for each memory mapping is already provided in the smaps file. Here's a command to calculate the total PSS value in kilobytes:

grep -e '^Pss' /proc/[PID]/smaps | awk '{sum += $2} END {print sum, "kB"}'

Replace [PID] with the actual process ID. The grep command filters the lines that start with 'Pss', and the awk command calculates the sum of these PSS values for all memory mappings.

To display the PSS value in megabytes, you can modify the command as follows:

grep -e '^Pss' /proc/[PID]/smaps | awk '{sum += $2} END {print sum / 1024, "MB"}'

Again, replace [PID] with the actual process ID. The awk command will sum the PSS values and convert the result to megabytes by dividing the sum by 1024.

For Example:

# grep -e '^Pss' /proc/16411/smaps | awk '{sum += $2} END {print sum / 1024, "MB"}'
1889.71 MB

The /proc/[PID]/smaps file also provides other memory-related information for each mapping, such as:

• Size: The total size of the mapping.
• Pss: The proportional set size (PSS), which is an estimation of the process's share of the mapping, considering the shared memory.
• Shared_Clean and Shared_Dirty: The clean and dirty (modified) shared memory used by the mapping.
• Private_Clean and Private_Dirty: The clean and dirty (modified) private memory used by the mapping.
• Referenced: The amount of memory currently marked as referenced or accessed.
• Anonymous: The amount of memory that doesn't belong to any file.

 

Why we prefer /proc/[PID]/smaps over other tools for checking memory usage per process?

Using /proc/[PID]/ can be preferred over other methods for checking memory usage in some cases because it provides more detailed and granular information about the memory usage of a process, including different types of memory such as shared, private, clean, and dirty memory. This level of detail allows you to obtain a better understanding of how a process is using memory and how much memory is actually attributable to that process.

Some reasons to prefer /proc/[PID]/smaps over other methods are:

1. Shared memory accounting: /proc/[PID]/smaps allows you to calculate PSS (Proportional Set Size), which accounts for shared memory among processes. PSS provides a more accurate representation of memory usage across multiple processes by avoiding double-counting shared memory. Other methods like top and ps might not provide PSS information directly.
2. Granularity: /proc/[PID]/smaps breaks down memory usage into individual memory mappings, showing how much memory is used by each mapping. This granularity can be useful for in-depth analysis or troubleshooting memory-related issues.
3. Flexibility: /proc/[PID]/smaps provides various memory-related metrics like RSS (Resident Set Size), USS (Unique Set Size), and PSS. Depending on your use case, you can choose the most appropriate metric to analyze memory usage.

However, there are also some drawbacks to using /proc/[PID]/smaps:

1. Complexity: Parsing and analyzing /proc/[PID]/smaps output can be more complex compared to using tools like top, ps, or htop, which provide simpler and more human-readable summaries of memory usage.
2. Performance: Reading and processing /proc/[PID]/smaps can be more resource-intensive, especially for processes with a large number of memory mappings.

 

Method-5: Using smem tool

smem is a command-line tool that provides memory usage information for processes, including PSS (Proportional Set Size) and USS (Unique Set Size) metrics. smem accounts for shared memory, which can give you a more accurate representation of memory usage across multiple processes.

The project page for smem is https://www.selenic.com/smem.

To use smem, you need to install it first. On Debian/Ubuntu-based systems, you can install smem using the following command:

sudo apt-get install smem

There are various filters which you can apply with smem as shown below with the latest available release (1.4) at the time of writing this article

# ./smem --help
Usage: smem [options]

Options:
  -h, --help            show this help message and exit
  -H, --no-header       disable header line
  -c COLUMNS, --columns=COLUMNS
                        columns to show
  -t, --totals          show totals
  -R REALMEM, --realmem=REALMEM
                        amount of physical RAM
  -K KERNEL, --kernel=KERNEL
                        path to kernel image
  -m, --mappings        show mappings
  -u, --users           show users
  -w, --system          show whole system
  -P PROCESSFILTER, --processfilter=PROCESSFILTER
                        process filter regex
  -M MAPFILTER, --mapfilter=MAPFILTER
                        map filter regex
  -U USERFILTER, --userfilter=USERFILTER
                        user filter regex
  -n, --numeric         numeric output
  -s SORT, --sort=SORT  field to sort on
  -r, --reverse         reverse sort
  -p, --percent         show percentage
  -k, --abbreviate      show unit suffixes
  --pie=PIE             show pie graph
  --bar=BAR             show bar graph
  -S SOURCE, --source=SOURCE
                        /proc data source

To check memory usage per process in total we can execute below command

# ./smem -t -k
  PID User     Command                         Swap      USS      PSS      RSS
 5496 ssrun    ssRelay -pidfile /opt/mgtse        0   112.0K   152.0K   460.0K
 5522 ssrun    ssProbeframework -pidfile /        0   116.0K   156.0K   464.0K
 2072 root     rhnsd                              0   172.0K   194.0K   616.0K
 2013 root     /usr/bin/rhsmcertd                 0   172.0K   195.0K   684.0K
 1617 root     /sbin/agetty --noclear tty1        0   184.0K   196.0K   876.0K
 1493 root     /usr/sbin/atd -f                   0   260.0K   271.0K     1.1M
22058 root     /usr/sbin/rpc.idmapd               0   380.0K   384.0K   676.0K
 7802 root     /usr/sbin/xinetd -stayalive        0   396.0K   408.0K     1.1M

<< Output trimmed >> 
 
22199 sufuser  /usr/sbin/httpd -DFOREGROUN        0   372.0K   769.0K     5.4M
27327 sufuser  /usr/sbin/httpd -DFOREGROUN        0   372.0K   769.0K     5.4M
22201 sufuser  /usr/sbin/httpd -DFOREGROUN        0   380.0K   775.0K     5.4M
22200 sufuser  /usr/sbin/httpd -DFOREGROUN        0   380.0K   776.0K     5.4M
12096 root     -bash                              0   580.0K   809.0K     2.3M
 7989 ntp      /usr/sbin/ntpd -u ntp:ntp -        0   780.0K   814.0K     1.7M
11564 oamsys   /usr/libexec/openssh/sftp-s        0   744.0K   834.0K     2.4M
 5523 ssrun    ssProbeframework -pidfile /        0   185.3M   191.3M   198.4M
-------------------------------------------------------------------------------
   93 11                                          0     1.9G     2.0G     2.1G

This will give you memory usage detail of every process active on your system.

To get the memory usage of a single process we can grep the process from the list

# ./smem -k | sed -e '1p' -e '/amsHelper/!d' | grep -v sed
  PID User     Command                         Swap      USS      PSS      RSS
31768 root     /sbin/amsHelper -f                 0    56.0M    56.4M    58.7M

Here as you see we get similar results as above with /proc/31768/smaps and with smem tool. So the actual memory usage of amsHelper is 56.4 MB

 

Summary

In Linux, various tools can be used to check memory usage per process, each providing different levels of detail and granularity.

1. ps: A versatile tool for displaying information about active processes. It provides basic memory usage information, such as RSS (Resident Set Size), but does not account for shared memory. It is suitable for quick overviews of memory usage across multiple processes, with options to customize the output format.
2. top: A real-time, interactive process monitor that shows memory usage along with other system metrics. It provides a convenient way to monitor memory usage across multiple processes simultaneously. However, it might not provide detailed memory usage metrics like PSS (Proportional Set Size) or USS (Unique Set Size) directly.
3. /proc/[PID]/smaps: A detailed source of memory usage information for a specific process, including different types of memory (shared, private, clean, dirty). It allows you to calculate PSS and USS, accounting for shared memory usage. The granularity of /proc/[PID]/smaps makes it suitable for in-depth analysis or troubleshooting memory-related issues.
4. smem: A command-line tool that provides memory usage information for processes, including PSS and USS metrics. It accounts for shared memory, giving a more accurate representation of memory usage across multiple processes. smem is useful when you need a more comprehensive view of memory usage that considers shared memory.
5. pmap: A tool that provides information about a process's memory mappings, including memory usage, permissions, and mapped files. pmap can help you understand how a process uses memory, but it does not account for shared memory or provide metrics like PSS or USS directly.

Choosing the right tool to check memory usage per process in Linux depends on your needs. For quick overviews or real-time monitoring, ps and top are convenient options. For detailed analysis and shared memory accounting, /proc/[PID]/smaps and smem provide more accurate metrics. To inspect memory mappings and their usage, pmap can be a useful tool.