Task ‘Health-Check operations for SDDC’ Fails in VCF SDDC Manager due to concurrent workflows

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

VCF Ops triggers the SOS Health Check API once daily to monitor health metrics from the VCF SDDC Manager.

If any workflows are running simultaneously, the task 'Health-Check operations for SDDC' may fail in VCF SDDC Manager, with a sub-task failure in Pre-validation [Validate Source, Validate Disk Space, Verify Inventory, Verify Active Workflows]

Environment

VCF Operations 9.0

VMware Cloud Foundation 5.x

Cause

The VCF SDDC Manager SOS API (https://<VCF_SDDC_MANAGER_HOST>/v1/system/health-summary) includes a guardrail that verifies if any workflows are currently running in the system.

If active workflows are detected, the request fails during pre-validation.

Resolution

This is a known issue in VCF 9.0 and can be ignored. A fix is planned for a future update.

If this issue is observed for VMware Cloud Foundation 5.x, please follow the resolution listed in the KB article here - https://knowledge.broadcom.com/external/article/324027/found-active-workflow-for-other-descript.html

Additional Information

VCF Operations sends these requests four times per domain, leading to an increased number of failed tasks.

The retry mechanism can be adjusted using the following adapter properties on all nodes.

You can reduce the retry count to minimize the number of failed tasks.

You need to modify the properties on all nodes and cloud proxies of VCF Opeartions cluster.

Properties file location: /usr/lib/vmware-vcops/user/plugins/inbound/VMwareInfrastructureHealthAdapter/conf/VMwareInfraHealth.properties

VCF_SOS_API_CALL_RETRY_COUNT = 4
VCF_SCHEDULED_THREAD_COLLECTION_DURATION_IN_MINS‎ =  1440