What are transparent hugepages? How to turn off or disable transparent hugepages in RHEL 7 Linux?

In my last article I shared the steps to improve disk IO performance in Linux. Now another important part of optimization of databases are the huge pages.

How to use, monitor disable transparent hugepages ( CentOS / RHEL 7 )

Huge pages are pages with a default size of two megabytes or more specific. Red Hat is using two different huge page sizes, the defaults are two megabytes and there’s an option to use bigger huge pages as well with the size of one gigabyte.

To get the page size use below commands

# getconf PAGE_SIZE
4096

# getconf PAGESIZE
4096

 

Why to use Hugepages?

  • One of the reasons can be huge pages help optimizing TLB.
  • TLB stands for Translation Lookaside Buffer.
  • A TLB is a memory cache that is used to reduce the time taken to access a user memory location.
  • This buffer is kept to keep information about huge memory pages into cache
  • This buffer isn’t very big and it needs precious memory so you want it to be as efficient as possible.
  • And that is why if you want to optimize the TLB usage using huge pages might be an option.

 

How to monitor Hugepages?

The number of anonymous transparent huge pages currently used by the system is available by reading the AnonHugePages field in /proc/meminfo

# grep -i AnonHugePages /proc/meminfo
AnonHugePages: 1216512 kB

To identify what applications are using anonymous transparent huge pages, it is necessary to read /proc/PID/smaps and count the AnonHugePages fields for each mapping.

# grep -e AnonHugePages /proc/$(pgrep test.sh)/smaps | awk '{ if($2>0) print $0} '
AnonHugePages: 120832 kB
NOTE:
There are a number of counters in /proc/vmstat that may be used to monitor how successfully the system is providing huge pages for use.

 

How to allocate Huge Pages?

You can allocate hugepae on runtime from the command line using “sysctl -w“. Now before making the reservation let us validate our hugepage reservation

# grep -i huge /proc/meminfo
AnonHugePages:     10240 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

So there are no reservation for hugepages, below are the available and used memory details

# free -m
              total        used        free      shared  buff/cache   available
Mem:           3790         194        3318          46         277        3314
Swap:           759           0         759
IMPORTANT NOTE:
This will only work if contiguous memory is available so you may have to do this right on startup of your machine because otherwise you’ll risk running out of contiguous memory.

Let us reserve 512MB for Huge Pages

# sysctl -w vm.nr_hugepages=512
vm.nr_hugepages = 512

If you observe immediately some part of our memory which was earlier free is now not available any more and they are reserved for hugepages

# free -m
              total        used        free      shared  buff/cache   available
Mem:           3790        1220        2292          46         277        2289
Swap:           759           0         759

Validate the hugepage reservation again

# grep -i huge /proc/meminfo
AnonHugePages:     14336 kB
HugePages_Total:     512
HugePages_Free:      512
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

To make the changes permanent, add these values to sysctl. I will create a new file “10-hugepages.conf” under “/etc/sysctl.d/”

# cat /etc/sysctl.d/10-hugepages.conf
vm.nr_hugepages=512
IMPORTANT NOTE:
When working with huge pages you should always notice that huge pages are no longer available as general memory so if you use it for huge pages you cannot use it for anything else.

 

How to disable Transparent Hugepages permanently using GRUB2?

To disable THP append “transparent_hugepage=never” to the kernel command line as shown below under /etc/sysconfig/grub file

GRUB_CMDLINE_LINUX="novga console=ttyS0,115200 panic=1 numa=off elevator=cfq rd.md.uuid=d265dd3d:9ee4d53a:597b8c08:8201b9af rd.lvm.lv=os/root rd.md.uuid=5398452a:ab1b8e91:4307b53b:5c3cccbd rd.md.uuid=131bc1e7:7c9087c3:03f3ad4a:7cde170c noht rhgb quiet transparent_hugepage=never"

Next rebuild your grub2 configuration file

# grub2-mkconfig -o /boot/grub2/grub.cfg

Reboot the node for the changes to take affect. Once the node is up make sure the newly added entry exists in your loaded grub configuration

# grep transparent_hugepage /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-862.6.3.el7.x86_64 root=/dev/mapper/os-root ro novga console=ttyS0,115200 panic=1 numa=off elevator=cfq rd.md.uuid=d265dd3d:9ee4d53a:597b8c08:8201b9af rd.lvm.lv=os/root rd.md.uuid=5398452a:ab1b8e91:4307b53b:5c3cccbd rd.md.uuid=131bc1e7:7c9087c3:03f3ad4a:7cde170c noht biosdevname=0 net.ifnames=0 rhgb quiet transparent_hugepage=never

So the configuration is correctly loaded.

 

How to disable Transparent Hugepages on runtime?

nr_hugepages indicates the current number of “persistent” huge pages in the kernel’s huge page pool. “Persistent” huge pages will be returned to the huge page pool when freed by a task.

When multiple huge page sizes are supported, /proc/sys/vm/nr_hugepages indicates the current number of pre-allocated huge pages of the default size.

Thus, one can use the following command to dynamically allocate/deallocate default sized persistent huge pages. If the value in /proc/sys/vm/nr_hugepages file or vm.nr_hugepages in sysctl.conf is “0” it means HugePages is disabled on the system

# echo 0 > /proc/sys/vm/nr_hugepages
# cat /proc/sys/vm/nr_hugepages
0
# sysctl vm.nr_hugepages
vm.nr_hugepages = 0

 

How to disable Transparent Hugepages permanently using sysctl?

To disable the explicit hugepage permanently add below entry in “/etc/sysctl.conf

# grep -i nr_hugepages /etc/sysctl.conf
vm.nr_hugepages=0

Refresh the values

# sysctl -p

Below command will show the updated value

# sysctl -a | grep nr_hugepages
vm.nr_hugepages = 0
NOTE:
A reboot is required to activate the changes.

 

Lastly I hope the steps from the article to disable transparent hugepages on Linux was helpful. So, let me know your suggestions and feedback using the comment section.

1 Comments

Leave a Reply

Your email address will not be published. Required fields are marked *