Aria Automation loses connectivity to vCenter Server and we can see following error in Aria Cloud Account UI "The connection with vAPI endpoint can not be established"

Products

• VMware vCenter Server
• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Aria Automation may lose connectivity to vCenter Server regularly 
• In Cloud Accounts you may see error like below: 

"java.util.concurrent.CompletionException: com.vmware.photon.controller.model.adapters.vsphere.VsphereAdapterException: The connection with vAPI endpoint can not be established. Cause: null "

• In vCenter Server /var/log/vmware/vapi/endpoint/endpoint.log  we may see errors similar to below: 

####-##-##T##:##:##.###Z | ERROR | vAPI-I/O dispatcher-0     | SessionFacade                  | ########################## | Unexpected error occurred while executing the call with session [email protected] (internal id #######) for method com.vmware.appliance.vmon.service.get.
com.vmware.vapi.client.exception.TransportProtocolException: HTTP response with status code 503 (enable debug logging for details): no healthy upstream
        at com.vmware.vapi.internal.protocol.client.rpc.http.ApacheHttpUtil.validateHttpResponse(ApacheHttpUtil.java:101) ~[vapi-runtime.jar:?]
        at com.vmware.vapi.internal.protocol.client.rpc.http.handle.NioSingleResponseConsumer.responseCompleted(NioSingleResponseConsumer.java:61) [vapi-runtime.jar:?]

• and/or 

[####-##-##T##:##:##.###Z ] [ERROR] io-127.0.0.1-####-exec-##### ######## ###### ####### com.vmware.vise.vim.commons.vcservice.impl.VcServiceImpl          Failed to retrieve the instance name of VC https:// VC-FQDN:443/sdk. Will use the URL's host as instance name java.util.concurrent.ExecutionException: (vim.fault.NoPermission) {  
   faultCause = null,  

• /var/log/vmware/sso/vmware-identity-sts.log may show errors like below: 

####-##-##T##:##:##.###Z ERROR sts[##:tomcat-http--10] [CorId=#############################] [com.vmware.identity.idm.server.IdentityManager] Failed to authenticate principal [user@domain] for tenant [vsphere.local] javax.security.auth.login.LoginException: Login failed
   

Environment

vCenter Server 8.0 

Aria Automation

Cause

STS certificates chain is incorrect or certificate may be expired

Resolution

Please follow following article steps to renew STS certificate: "Signing certificate is not valid" or "No healthy upstream" error in vCenter Server Appliance

After renewal 1 Leaf certificate and 1 Root certificate should be present in the chain 

Monitor Aria environment to confirm no further connectivity issues observed