UI sluggishness after receiving the alert "vRealize Operations Cluster processes may not have enough memory"

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• After receiving the administrative alert 'vRealize Operations Cluster processes may not have enough memory' access to the Aria Operations UI becomes very slow and the node starts consuming a lot of CPU.
• This may resolve after some time or may require that the cluster is brought down and up to remediate.
• During these times, the metric 'Garbage collection GC workload (%)' will spike to 100%, and the metric 'Elastic Heap Memory Remaining (%)' drops to 5% for the Aria Operations master nodes object.
• Heapdump files may be generated by these events on the Aria Operations master node.
• The cluster in CA-Enabled
• After some time Aria Operation node starts consuming too much CPU and leaves the cluster inoperative.
• The Garbage Collection | GC Workload (%) shows high on affected node(s).

Environment

Aria Operations 8.18.x

Cause

This is a known bug with the version of Gemfire is use on this version of Aria Operations.

It has been predominantly reported in CA clusters where the fault domains are in geographically different locations.

Gemfire is extremely sensitive to high latency and this is the root of the problem.

Resolution

There is no direct resolution for this at the moment.

A newer version of Gemfire will be in use for VCF 9, where this bug is not present.