A VMware Tanzu GemFire service instance goes into read-only mode (put operations fail), when heap consumption crosses the critical threshold

Products

• VMware Tanzu Gemfire

Issue/Introduction

A VMware Tanzu GemFire service instance goes into read-only mode (put operations fail), when heap consumption crosses the critical Threshold. Normal operations are only fully restored once your tenured heap consumption goes back down to a more healthy level below the critical-threshold. This reduction is driven by the fact that data is getting evicted, creating garbage, and allowing GC to run and reduce that tenured heap utilization.

This is applicable to all versions of VMware Tanzu GemFire. The critical threshold is set to either 90% or 95% of tenured heap, depending on the VMware Tanzu GemFire plan being used. This is a feature to prevent data inconsistency and crashes, when the heap consumption crosses the critical threshold. When VMware Tanzu GemFire goes into a read-only mode, gets will still work while queries and puts will fail.

If the system stays for too long in this read-only mode, the membership of the system will likely be impacted, such as members getting kicked out of the cluster.  Thus, monitoring of your cluster is essential to avoid hitting these levels of consumption.

Resolution

The recommendation is to proactively monitor the heap usage using service metrics and then take corrective actions like scaling horizontally or vertically.

 

VMware Tanzu GemFire clusters and brokers emit service metrics. Any tool that has a corresponding Cloud Foundry nozzle to read and monitor these metrics in real time, can be used to generate events or alerts.

The metric serviceinstance.UsedHeapSize can be used to monitor the heap usage and generate an event or an alert.
​​
For implementation details, please refer to the Monitoring Pivotal Cloud Cache Service Instances, section in the VMware Tanzu GemFire Docs: https://docs.pivotal.io/p-cloud-cache/1-9/monitoring.html