GemFire: Using ZGC requires more individual memory mappings from OS than other algorithms

Products

• VMware Tanzu Gemfire

Issue/Introduction

Customers using ZGC may experience issues with GemFire members crashing for unknown reasons.    It may or may not be Java 17 specific, but regardless, it is important to understand that the internals of ZGC drive a requirement to have a greater number of memory mappings than older/other collectors

Please examine GC logs prior to opening a Support ticket with GemFire if using ZGC.  If you see the below warning, take the appropriate action to adjust your /proc/sys/vm/max_map_count as instructed: 

[0.389s][warning][gc] ***** WARNING! INCORRECT SYSTEM CONFIGURATION DETECTED! *****
[0.389s][warning][gc] The system limit on number of memory mappings per process might be too low for the given
[0.389s][warning][gc] max Java heap size (208896M). Please adjust /proc/sys/vm/max_map_count to allow for at
[0.389s][warning][gc] least 376012 mappings (current limit is 65530). Continuing execution with the current
[0.389s][warning][gc] limit could lead to a premature OutOfMemoryError being thrown, due to failure to map memory.
[0.639s][info   ][gc] Using The Z Garbage Collector
[1.822s][info   ][gc] GC(0) Garbage Collection (Metadata GC Threshold) 104M(0%)->56M(0%)
[2.788s][info   ][gc] GC(1) Garbage Collection (Metadata GC Threshold) 134M(0%)->60M(0%)
[6.777s][info   ][gc] GC(2) Garbage Collection (Metadata GC Threshold) 290M(0%)->82M(0%)
[21.945s][info   ][gc] GC(3) Garbage Collection (Warmup) 21132M(10%)->6692M(3%)
[41.444s][info   ][gc] GC(4) Garbage Collection (Warmup) 42010M(20%)->28038M(13%)
[44.503s][info   ][gc] GC(5) Garbage Collection (Allocation Rate) 28724M(14%)->19178M(9%)
[49.828s][info   ][gc] GC(6) Garbage Collection (Allocation Rate) 39730M(19%)->18958M(9%)
[59.861s][info   ][gc] GC(7) Garbage Collection (Warmup) 62926M(30%)->20370M(10%)
[64.124s][info   ][gc] GC(8) Garbage Collection (Allocation Rate) 26960M(13%)->34886M(17%)
[350.866s][info   ][gc] GC(9) Garbage Collection (Proactive) 157490M(75%)->10204M(5%)
[704.144s][info   ][gc] GC(10) Garbage Collection (Allocation Rate) 50468M(24%)->25554M(12%)
[1282.706s][info   ][gc] GC(11) Garbage Collection (Proactive) 110070M(53%)->13642M(7%)

Environment

Product Version: 9.15
OS: All, JDK 17+

Resolution

Follow the above warning instructions, and potentially give yourself some additional headroom beyond suggested max_map_count.

Please work with appropriate resources to set this property and to make sure it is persisted across restarts.