Heap size grows to higher levels because of auto-reconnect in GemFire

Products

• VMware Tanzu Gemfire

Issue/Introduction

You are experiencing higher levels of heap in GemFire that are not fully understood. You may be experiencing higher CPU, eviction, or critical heap processing due to the heap growing above expected healthy levels of consumption.

This issue could be related to a leak related to a member having been auto-reconnected at some point since the last full restart.

Environment

Product Version: 9.1
OS: All

Resolution

This issue is resolved in the latest versions of GemFire 9.10.8 and above.

"GEODE-9307, GEM-3293: Corrected increased heap consumption following auto-reconnection caused by region references that should have been destroyed following a forced disconnect."

Workaround

If you are unable to upgrade at this time, it will be necessary to perform a full bounce of any cache server that has been auto-reconnected since its last full restart. Until this workaround is applied, the heap will be higher, and potentially drive the cluster to be unstable.

Perform a full bounce of each cache server impacted. Only perform a full bounce on a cache server one at a time to preserve full access to data. This restores the heap to lower expected levels.

This is necessary each time a member is kicked out and auto-reconnects.

Note: It is important to have enough monitoring to know when a member is auto-reconnected. We recommend keeping on eye on whether there were any changes in membership over time.