Gemfire: Failure to reserve large pages memory (errno = 12)

Products

• VMware Tanzu Gemfire

Issue/Introduction

Gemfire server fails to start with errno = 12 error as shown below

........The Cache Server process terminated unexpectedly with exit status 1. 

OpenJDK 64-Bit Server VM warning: Failed to reserve large pages memory req_addr: 0x0000000000000000 bytes: 165356240896 (errno = 12).

OpenJDK 64-Bit Server VM warning: Failed to reserve large pages memory req_addr: 0x00007f8abbc00000 bytes: 2279604224 (errno = 12).

 

Cause

Error (errno = 12) indicates that the Java Virtual Machine (JVM) tried to allocate memory using Large Pages (also known as HugePages in Linux), but the attempt failed.

 

Resolution

To resolve this error, change OS level configurations in the order listed below.

 

1) Change memlock ulimit values in /etc/security/limits.conf which is most likely to resolve startup issues

soft memlock unlimited

hard memlock unlimited

2) Disable Transparent Huge Pages (THP) as mentioned in the KB.

To verify if THP is enabled run the command, cat /proc/meminfo | grep Huge

In the example below, the last entry shows that TLB is enabled.

AnonHugePages:  78972928 kB

ShmemHugePages:    0 kB

FileHugePages:     0 kB

HugePages_Total: 1024

HugePages_Free:  834

HugePages_Rsvd:   97

HugePages_Surp:    0

Hugepagesize:   2048 kB

Hugetlb:     2097152 kB

3) Improve virtualization

• • Disable hyperthreading
  • 100% reservation for cpu
  • 100% reservation for memory
  • Enable latency sensitivity
  • Reserve 4-8 physical cores on each host for hypervisor / IO requirements

4) Improve swap

         Refer to the article on configuring swappiness in different Redhat versions.  A related KB explains vm swapiness and it's effect on Gemfire.

  

 

Additional Information

For any issues related to this error, please collect the information listed below when you raise a Support case.

1) ulimit -a

2) grep -i hugepages /proc/meminfo

3) dmesg | grep -Ei "huge|error|fail"

4) grep -Ei "huge|error|fail" /var/log/syslog (also zgrep -Ei "huge|error|fail" /var/log/syslog* if the log has rotated)

5) grep -R . /sys/kernel/mm/hugepages/

6) cat /proc/sys/vm/nr_hugepages