GemFire: Low Memory Exception in Gateway Receiver Due to ZGC Memory Usage Calculation

Products

• VMware Tanzu Gemfire

Issue/Introduction

When transferring data into one cluster for a specific region, the Gateway Receiver at another cluster may encounter Low Memory exceptions. This occurs as the Gateway Receiver server reaches the critical memory threshold.

 

Environment

• Versions prior to 9.15.13, 10.0.5 and 10.1.2

• Using ZGC as the garbage collector

• Gateway Receiver configured for data transfer between clusters

• Resource Manager is set

Cause

• The Resource Manager currently determines eviction and critical threshold decisions based on currentUsage of the ZHeap memory pool.

• In ZGC, currentUsage includes both short-lived and long-lived objects. A rapid burst of short-lived allocations can artificially increase memory usage, triggering unnecessary evictions and Low Memory exceptions.

• Unlike CMS, where currentUsage was appropriate due to the tenured memory pool, ZGC operates differently with a single memory pool (ZHeap).

• A better approach is to use collectionUsage, which accounts for memory usage after ZGC completes a collection, reducing unnecessary evictions.

 

Resolution

1. Upgrade to a Fixed Version:

   • Upgrade to GemFire 9.15.13, 10.0.5 or 10.1.2,  which includes a fix for this issue by allowing the use of collectionUsage instead of currentUsage for ZGC.

2. Avoid Using Resource Manager:

   • If upgrading is not immediately possible, consider disabling the Resource Manager to prevent it from making decisions based on currentUsage.

3. Increase Critical Threshold:

   • If the load size is expected to remain stable, increasing the critical threshold can help mitigate the issue.