How to check and disable transparent hugepages ( CentOS / RHEL 7 )

In my last article I shared the steps to improve disk IO performance in Linux. Now another important part of optimization of databases are the Transparent HugePages. So in this article I will share the steps to disable transparent hugepages with examples for runtime and to disable them permanently (reboot persistent).

Huge pages are pages with a default size of two megabytes or more specific. Red Hat is using two different huge page sizes, the defaults are two megabytes and there's an option to use bigger huge pages as well with the size of one gigabyte.

Why to use Hugepages?

• One of the reasons can be huge pages help optimizing TLB.
• TLB stands for Translation Lookaside Buffer.
• A TLB is a memory cache that is used to reduce the time taken to access a user memory location.
• This buffer is kept to keep information about huge memory pages into cache
• This buffer isn't very big and it needs precious memory so you want it to be as efficient as possible.
• And that is why if you want to optimize the TLB usage using huge pages might be an option.

How to monitor Transparent Hugepages?

The number of anonymous transparent huge pages currently used by the system is available by reading the AnonHugePages field in /proc/meminfo

# grep -i AnonHugePages /proc/meminfo
AnonHugePages: 1216512 kB

To identify what applications are using anonymous transparent huge pages, it is necessary to read /proc/PID/smaps and count the AnonHugePages fields for each mapping.

# grep -e AnonHugePages /proc/$(pgrep test.sh)/smaps | awk '{ if($2>0) print $0} '
AnonHugePages: 120832 kB

How to allocate HugePages?

You can allocate hugepages on runtime from the command line using "sysctl -w". Now before making the reservation let us validate our hugepage reservation

# grep -i huge /proc/meminfo
AnonHugePages:     10240 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

So there are no reservation for hugepages, below are the available and used memory details

# free -m
              total        used        free      shared  buff/cache   available
Mem:           3790         194        3318          46         277        3314
Swap:           759           0         759

Let us reserve 512MB for Huge Pages

# sysctl -w vm.nr_hugepages=512
vm.nr_hugepages = 512

If you observe immediately some part of our memory which was earlier free is now not available any more and they are reserved for hugepages

# free -m
              total        used        free      shared  buff/cache   available
Mem:           3790        1220        2292          46         277        2289
Swap:           759           0         759

Validate the hugepage reservation again

# grep -i huge /proc/meminfo
AnonHugePages:     14336 kB
HugePages_Total:     512
HugePages_Free:      512
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

To make the changes permanent, add these values to sysctl. I will create a new file "10-hugepages.conf" under "/etc/sysctl.d/"

# cat /etc/sysctl.d/10-hugepages.conf
vm.nr_hugepages=512

How to disable Transparent Hugepages (AnonHugePages) on runtime?

To disable Transparent Hugepages for the current session, execute the below command. This will stop only creation and usage of the new THP.

# echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled

OR (depending upon your distro)

# echo never > /sys/kernel/mm/transparent_hugepage/enabled

Next you can verify the status of transparent hugepages (AnonHugePages).

How to disable Transparent HugePages (AnonHugePages) permanently?

To disable Transparent HugePages (AnonHugePages) permanently, append "transparent_hugepage=never" to the kernel command line as shown below under /etc/sysconfig/grub file

GRUB_CMDLINE_LINUX="novga console=ttyS0,115200 panic=1 numa=off elevator=cfq rd.md.uuid=d265dd3d:9ee4d53a:597b8c08:8201b9af rd.lvm.lv=os/root rd.md.uuid=5398452a:ab1b8e91:4307b53b:5c3cccbd rd.md.uuid=131bc1e7:7c9087c3:03f3ad4a:7cde170c noht rhgb quiet transparent_hugepage=never"

Next rebuild your GRUB2 configuration file

# grub2-mkconfig -o /boot/grub2/grub.cfg

Reboot the node for the changes to take affect.

# shutdown -r now

Once the node is up make sure the newly added entry exists in your loaded grub configuration

# grep transparent_hugepage /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-862.6.3.el7.x86_64 root=/dev/mapper/os-root ro novga console=ttyS0,115200 panic=1 numa=off elevator=cfq rd.md.uuid=d265dd3d:9ee4d53a:597b8c08:8201b9af rd.lvm.lv=os/root rd.md.uuid=5398452a:ab1b8e91:4307b53b:5c3cccbd rd.md.uuid=131bc1e7:7c9087c3:03f3ad4a:7cde170c noht biosdevname=0 net.ifnames=0 rhgb quiet transparent_hugepage=never

So the configuration is correctly loaded.

Next you can verify the status of transparent hugepages (AnonHugePages)

How to disable Explicit Transparent Hugepages (nr_hugepages) on runtime?

nr_hugepages indicates the current number of "persistent" huge pages in the kernel's huge page pool. "Persistent" huge pages will be returned to the huge page pool when freed by a task. A user with root privileges can dynamically allocate more or free some persistent huge pages by increasing or decreasing the value of 'nr_hugepages'.

When multiple huge page sizes are supported, /proc/sys/vm/nr_hugepages indicates the current number of pre-allocated huge pages of the default size.

Thus, one can use the following command to dynamically allocate/deallocate default sized persistent huge pages. If the value in /proc/sys/vm/nr_hugepages file or vm.nr_hugepages in sysctl.conf is "0" it means HugePages is disabled on the system

To disable explicit transparent hugepages for the current session execute below command:

# echo 0 > /proc/sys/vm/nr_hugepages

# cat /proc/sys/vm/nr_hugepages
0

# sysctl vm.nr_hugepages
vm.nr_hugepages = 0

Next you can verify the status of explicit transparent hugepages (nr_hugepages) on your system.

How to disable Explicit Transparent Hugepages (nr_hugepages) permanently using sysctl?

Now let me guide you with the steps to disable explicit transparent hugepages permanently. I will create a new sysctl file and add vm.nr_hugepages=0

# cat /etc/sysctl.d/disable_hugepage.conf
vm.nr_hugepages=0

Now before I refresh dracut entry, let us verify the initramfs content to make sure it does not has hugepage entry from sysctl.

# lsinitrd /boot/initramfs-`uname -r`.img | grep hugepage

Now let us refresh our dracut entry

# dracut -f

Now re-verify the initramfs content. As you see our new sysctl content is available now.

# lsinitrd /boot/initramfs-`uname -r`.img | grep hugepage
-rw-r----- 1 root root 18 Jul 31 18:46 etc/sysctl.d/disable_hugepage.conf

Verify the status of explicit transparent hugepage

# cat /sys/devices/system/node/node*/meminfo | fgrep Huge
Node 0 AnonHugePages: 0 kB
Node 0 HugePages_Total: 0
Node 0 HugePages_Free: 0
Node 0 HugePages_Surp: 0

Next you can verify the status of explicit transparent hugepages (nr_hugepages) on your system.

Lastly I hope the steps from the article to disable transparent hugepages and allocate hugepages on Linux was helpful. So, let me know your suggestions and feedback using the comment section.