Gemfire: High SWAP memory usage causes cluster to go down.

Products

• VMware Tanzu Gemfire

Issue/Introduction

High SWAP memory usage causes  cluster to go down. 

Gemfire runs as a java process and there are times when it uses swap memory which can negatively affect its performance or destabilize the cluster. There are some JVM and Gemfire configurations that can prevent this scenario and these are listed below.

Environment

Gemfire 9.15

Resolution

1. Heap Settings: Set minimum (-Xms) heap equal to maximum (-Xmx) heap  across all environments and instances. 
2. Pre-touch Memory Allocation: Set the -XX:+AlwaysPreTouch JVM flag to force the JVM to pre-touch the memory pages it needs at startup, ensuring that all necessary RAM is allocated upfront. This would prevent RAM from being allocated to swap and would improve performance during normal operation by reducing page faults.  
3. Off-Heap Memory Usage: To prevent the use of swap space, the key recommendations from the documentation are:
• Configure the ulimit setting to allow for the maximum amount of memory( max offheap memory + max heap) your application might require. 
• When starting a GemFire member, use the gfsh start server ... -lock-memory=true command to lock the memory. This prevents the OS from swapping out memory, which is critical for maintaining consistent performance.
• To avoid potential race conditions on certain operating systems, which could result in failures or crashes, it is recommended to launch each JVM or GemFire instance sequentially when using memory locking. This approach ensures that each instance secures the necessary resources without contention.
• Optionally drop the  OS caches to reduce startup times. This step can help reduce the start up time especially since the steps mentioned above can increase the start up time.