GemFire: Using ZGC and SoftMaxHeapSize setting with Eviction

Products

• VMware Tanzu Gemfire

Issue/Introduction

How do I use the ZGC Garbage Collector with GemFire Heap Eviction?

Environment

Product Version: 9.15
OS: ALL, Java 17+

Resolution

Some brief background... Since Java 9, G1GC has been the default Garbage Collector for Java. But since G1GC presents unique challenges when used in conjunction with GemFire's heap LRU eviction feature, VMware recommends using other GC algorithms for most applications. See KB article #388242 Heap LRU Eviction With The G1 Garbage Collector if you want more details.

As of GemFire 9.15.x and going forward, GemFire is now compatible with Java 17, and has been validated using ZGC. Thankfully, ZGC is compatible with GemFire Eviction with some very simple configuration settings.

The instructions are very basic, and simply requires that you have a good understanding of your real heap consumption per member.

1. Determine your total expected heap consumption PER MEMBER, assuming balance across members. This involves sizing all of your keys,values across all regions and that piece of the puzzle is not part of this KB.
2. Based on that expected total per member, you should then determine the total heap needed per member, namely, your Xmx setting. This is the one most important setting for ZGC. Generally, I would recommend setting Xmx to 3 times your total expected heap consumption per member.
3. Based on our intensive ZGC validation within our GemFire team, we recommend setting the SoftMaxHeapSize to about 5% above your expected heap consumption, again per member.  Setting it above the expected heap consumption threshold based on your sizing should avoid spending too many cycles driving collections in general.
4. Finally, set the eviction-threshold (eviction-heap-percentage) about 3% above the SoftMaxHeapSize as a general rule, first attempt at tuning. Higher allocation rates, if a really busy system, may require the eviction-threshold to be even higher, perhaps 5% above the SoftMaxHeapSize.

Example:
Expected Heap Consumption Roughly:  20g
Xmx60g
SoftMaxHeapSize: 22g - 25g range depending on allocation rates
eviction-threshold: 25g - 30g range depending on allocation rates, minimum 3g above the SoftMaxHeapSize.

In the end, this all requires testing with Production like loads as much as possible. The goal is to avoid the need to evict as much as possible, first and foremost. That said, if some burst of activity in the system does drive heap consumption higher,  eviction should kick in to protect the system from hitting out of memory issues, allocation stalls, etc.

Final Note: GemFire eviction with overflow to disk evicts ONLY the values, not the keys or the GemFire entry overhead. Heap can continue to increase as the number of entries increases, even for a system configured to evict the values.