"java.lang.OutOfMemoryError" when running multiple GemFire server processes on node

Products

• VMware Tanzu Gemfire

Issue/Introduction

Out of Memory (OOM) errors in GemFire can be caused by:

1. An insufficient amount of free OS/System memory.
2. There are too many open files.
3. There are too many processes or threads for the user that’s logged in.
4. The system is unable to allocate enough memory for the stack.

If you are running multiple GemFire processes as the same user on the same node, having too many open files or too many processes or threads is likely the cause of the OOM error. This issue is caused when there is not enough soft ulimits for the open files (-n) and the processes (-u).

This is because soft ulimit is per user and if you are running multiple GemFire processes as the same user on the same node, the actual value is now divided by the number of the GemFire processes.

Environment

Product Version: 9.7

Resolution

These are the recommended values per user for Unix/Linux:

• The number of file descriptors (-n): The soft limit is 8192, and the hard limit is 81920.
• the number of processes (-u): The soft limit is 501408, with an unlimited hard limit.

Workaround

If you are running multiple GemFire processes as the same user on the same node, multiply the soft limit by the number of GemFire processes to prevent the OOM error.