Gemfire node fails with "Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN)..."

Products

• VMware Tanzu Data Intelligence

Issue/Introduction

Gemfire node fails with "Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 4k, detached."

Environment

All supported Gemfire versions

Cause

The error pthread_create failed (EAGAIN) for attributes: stacksize: xxxxk. EAGAIN  indicates that an OS per-process resource limit has been reached and is not the result of  exhausted memory.   In this scenario, the kernel returns an out-of-memory condition for that mapping, which the C library reports  as EAGAIN, and which the JVM in turn surfaces as OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached.

A large heap enabled with ZGC or Generational ZGC is more susceptible to this condition as its memory management strategy  can easily push past the Linux default limit of 65,530 for vm.max_map_count .

A second, lower-probability contributor would be a per-account thread limit (ulimit -u) ; the EAGAIN error is also consistent with that condition.

This specific failure can be distinguished from heap or memory problems by the following signature:

• The server log shows OutOfMemoryError: unable to create native thread together with pthread_create failed (EAGAIN) in the JVM diagnostics, while free RAM is high and the live thread count is normal (roughly 1,500 to 2,000).
• Hundreds of these errors appear within about one second, across unrelated components.
• "Please adjust /proc/sys/vm/max_map_count" message is present in the startup log.

 

Resolution

While restarting the nodes will provide immediate relief, the resolution is to increase vm.max_map_count.

To permanently raise vm.max_map_count:

Log onto each of your  cache server hosts as root and increase the mapping capacity. Since the max_map_count only represents the maximum limit and doesn't actively reserve memory from the OS, you can choose an upper limit as high as 2000000 for very large heaps. This provides ample headroom for large ZGC heaps and its active thread stacks.

1. Add or modify the configuration line inside /etc/sysctl.conf (or your configuration management directory /etc/sysctl.d/):

    

   vm.max_map_count=2000000 

2. Apply the setting immediately to the running kernel without a reboot:

    

   sysctl -w vm.max_map_count=2000000