Error while collecting data from vCenter cloud account on Aria Automation 8.x

Products

• VMware Aria Suite

Issue/Introduction

• Data collection on vCenter cloud account fails with error 

"java.util.concurrent.CompletionException: com.vmware.photon.controller.model.adapters.vsphere.VsphereAdapterException: The connection with vAPI endpoint can not be established. Cause: null" 

• Cloud account endpoint marked as suspended after the health check failed for a consecutive number of times. After fixing the issues related with the health check and revalidating the cloud account, enumeration continues to be skipped and doesn't run.
• Some event seen in provisioning logs:

provisioning-service-app.log

2024-08-29T07:55:52.740Z WARN provisioning [host='provisioning-service-app-##########-#####' thread='xn-index-queries-4' user='provisioning-###########' org='#########-####-####-####-############' trace='########-####-####-####-############' parent='########-####-####-####-############' span='#########-####-####-####-############'] c.v.xenon.common.ServiceErrorResponse.create:85 - message: Failed to send enum request. Endpoint [/resources/endpoints/#########-####-####-####-############] is in SUSPENDED state., statusCode: 400, serverErrorId: ########-####-####-####-############: Caused by null
provisioning-service-app.log.#:1781:"message": "Failed to send enum request. Endpoint [/resources/endpoints/###########-####-####-####-############] is in SUSPENDED state.", 

Environment

VMware Aria Automation 8.x

Cause

More than 10 consecutives connection failures to the endpoint due to expired vCenter password, vAPI Service issues or expired certificate.

Resolution

If assistance with this procedure is desired, please open a support request and reference this KB article.

NOTE: Take a non-memory snapshot of all the vRA nodes, also be sure that there are known good backups to restore from.

• Access the provisioning-db using this command:

# vracli dev psql provisioning-db

• Health check the endpoint resources with below query:

select document_self_link, name, custom_properties->>'consecutiveFailuresCount' as consecutiveFailuresCount, custom_properties->>'endpointHealthCheckState' as endpointHealthCheckState from endpoint_state;

• Find the endpoint which has reached 10 failures or is in SUSPENDED state, copy their value in the document_self_link column and use the following commands to reset their failures count to 0 and set the endpoint_state to AVAILABLE:

UPDATE endpoint_state SET custom_properties = jsonb_set(custom_properties, ' {consecutiveFailuresCount}
', '0', true) WHERE document_self_link = '<DOCUMENT_SELF_LINK_VALUE>';
UPDATE endpoint_state SET custom_properties = jsonb_set(custom_properties, '
{endpointHealthCheckState}
', '"AVAILABLE"', true) WHERE document_self_link = '<DOCUMENT_SELF_LINK_VALUE>';

     •  Re-validate & save the cloud account and wait for the data collection cycle to start.

Example: