vRealize Operations UI periodically becomes unresponsive

Products

VMware Aria Suite

Issue/Introduction

Symptoms:

The vRealize Operations Admin UI and/or Product UI are unresponsive
Data collections are not occurring
Alerts notifications are not sending
In the /storage/vcops/log/analytics-[uuid].log, you see entries similar to:

2022-04-03T14:25:57,306+0000 FATAL [Threshold checker worker thread 6] com.integrien.analytics.dataobject.AnalyticsEngine.updateExistingResourceCache - Gemfire serialization exception: toData failed on dsfid=71 msg:tcp message exceeded max size of 16777215

Note: Replace [uuid] with the unique identifier of your analytics node

Environment

VMware vRealize Operations 8.4.x
VMware vRealize Operations 8.6.x

Resolution

This is a known issue affecting vRealize Operations 8.4 and higher.

This issue is resolved in VMware Aria Operations 8.10 (build number 20570121).

Workaround:
To temporarily restore vRealize Operations functionality:

Log in to the primary vRealize Operations node as root via SSH or vSphere Console
Restart the vRealize Operations service:

systemctl restart vmware-vcops

Log in to the Admin UI at https://IPorFQDN/admin with the local admin account

Note: Replace IPorFQDN with the IP or FQDN of your vRealize Operations instance

Click the Take Cluster Offline button

Note: Wait for the Cluster Status to show "Offline"

Click the Bring Cluster Online button

Note: Due to the nature of this issue, it is not established how long this workaround will function, and the issue may recur.