GemFire: G1GC G1OldGeneration Collections are FULL GC's

Products

• VMware Tanzu Gemfire

Issue/Introduction

The goal of this KB is to help people troubleshoot GemFire issues when using G1GC.

If you see G1OldGeneration VMGCStats Collections, this is likely the major source of any issues, as this implies they have experienced a FULL GC when using G1.   

When troubleshooting GemFire issues where the customer is using G1GC, pay particular attention to the G1OldGeneration COLLECTIONS statistic.   

If there have been ANY, this means G1GC has done a FULL GC, which has/will likely drive GemFire members to be kicked out of the Distributed System due to being unresponsive.  

This is definitely the area to focus on, and use other KB articles to dig deeper into how to best configure and tune G1GC Heaps.  One such article is shared here:

GemFire: Using G1GC effectively, configuration and tuning needed

Symptoms:
Most likely symptom is that a member or members of GemFire are getting kicked out of the Distributed System.

You can look for JVM Pauses, log messages with the word "wakeup" which is normally the most specific log message indicating that JVM Pauses are occurring for GemFire JVMs.     If you see these "wakeup" messages, corresponding with use of G1GC, check for whether these G1OldGeneration collections are occurring.

In the G1GC logs, you may want to look to "to-space exhausted" and/or "Full GC" messages to help you confirm and get more specific details

Environment

Product Version: 9.10 and above
OS: ALL

Cause

Possible causes are not enough total heap, or some sub-optimal G1GC heap configuration that drives G1 to Pause the JVM to defragment G1 heap regions to create more room.   Such pauses will make JVM's unresponsive, and other members of GemFire may kick out such JVMs due to this lack of response, to keep the remaining healthy members functional.

Resolution

Add heap.   Tune heap.