VCF HRM Scripts take an excessively long time, fail to complete, or fail with a 500 error.

Products

• VMware SDDC Manager

Issue/Introduction

When using the Python Module for VMware Cloud Foundation Health Monitoring in VMware Aria Operations (v2.2.0.1003 or earlier) in a large environment (in excess of 15k VM's across multiple regions/ vCenters) you may see the following behavior:

• Data collection python scripts hang up for extended periods of time (e.g. Publish-VmConnectedCdrom does not return for 4+ hours)
• send-data-to-vrops.log shows repeated Polling again in 30 seconds for a long period of time but ultimately fails returning the error: INFO - [SosRest.py:144] - <Response [500]>

Environment

VMware Aria Operations 8.x

Cause

Issues in communications to SOS services and other performance related scripting updates needed.

Resolution

Please update the python module for HRM to v2.2.1 or higher.

Link - https://pypi.org/project/vmware-cloud-foundation-health-monitoring/ 

Update instructions - https://github.com/vmware-samples/validated-solutions-for-cloud-foundation/tree/main/hrm#updating-the-python-module-to-the-latest-version