GemFire: AEQ events cause memory spikes from G1 humongous allocations
Article ID: 420990
Updated On:
Products
Issue/Introduction
AEQ events trigger memory spikes in GemFire servers running G1GC, causing heap usage to exceed LRU eviction thresholds. This forces events to spill to disk, which generates additional large object allocations that become humongous regions, creating a feedback loop of increasing memory pressure and potential LowMemoryExceptions or cluster instability.
Example G1GC log indicating humongous allocation pressure:
[info][gc,heap] GC(xxxx) Humongous regions: yyyy->yyyy … Pause Young (Concurrent Start) (G1 Humongous Allocation)Environment
- All supported GemFire versions
- G1GC with Java17
Cause
Large objects on the heap are allocated as G1GC "humongous" objects, which occupy dedicated regions that cannot store other objects, causing inefficient heap usage and fragmentation. When heap usage exceeds the LRU eviction threshold, AEQ events spill to disk, triggering more humongous allocations in a feedback loop that drives memory spikes.
Resolution
Short-term fix:
Increase Java heap size (-Xmx) on each server so live data plus AEQ spikes stay below eviction/critical thresholds; test in a non-production environment first.
Permanent fix
Upgrade to Java 21 + Generational ZGC + Tanzu GemFire 10.1.4 (or later). This includes GEM-14595, which fixes heap utilization calculations for Heap LRU Eviction and critical thresholds under Generational ZGC.
Additional Information
Workaround
Restart affected GemFire servers (or the full cluster during maintenance) to clear humongous allocations and reset heap usage.
Generational ZGC Benefits
ZGC eliminates region-based fragmentation and treats all objects uniformly, including large ones—no humongous regions or wasted space. It uses virtual/physical memory decoupling for dense allocation and low-pause compaction.
Recommendations
- Review code for domain objects with huge collections, fat PDX-serialized objects, or large arrays that create ≥16 MB payloads.
- Enable app-level logging/metrics for region puts, value sizes, and AEQ traffic spikes.
- Check for session/event traffic surges driving large object allocations.
- Correlate G1GC logs (humongous allocation events) with VSD region stats to pinpoint culprit regions.
References:
VMware Tanzu GemFire 10.1 Release Notes
Feedback
