Intermittent Data Collection and Provisioning Failures in Aria Automation: "The connection with the vAPI endpoint cannot be established"

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Data collection intermittently enters a failed state for the vCenter cloud account.
  
  

• Operations such as provisioning new deployments, running Day-2 actions on existing deployments, validating cloud account credentials, or performing image synchronization intermittently fail.

• The Aria Automation UI displays the following error message: Failed to validate credentials. Error: The connection with vAPI endpoint can not be established. Please make sure the vAPI endpoint is running in the vCenter.

• Logs on Aria Automation located at /services-logs/prelude/provisioning-service-app/file-logs/provisioning-service-app.log show entries similar to:
  
  YYYY-MM-DDThh:mm:ss ERROR provisioning [host='provisioning-service-app-##########-#####' thread='OkHttp https://<VC-FQDN>/...' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.r.cis.VapiSession.lambda$login$3:254 - Failed during vAPI login. Exception: java.util.concurrent.CompletionException: java.net.SocketTimeoutException: timeout
Caused by: java.net.SocketTimeoutException: timeout
  
  
• You observe significant gaps in response times between the vpxd request and the sso response within the vCenter logs.

  
  /var/log/vmware/vpxd/vpxd.log (Request initiated):

YYYY-MM-DDThh:mm:ss info vpxd[#####] [Originator@#### sub=SsoClient opID=Run-Http2ServerSession-8] Successfully acquired token: SamlToken [subject={Name: <USer-Name>; Domain:<Domain-Name>}, groups=[...], startTime=YYYY-MM-DDThh:mm:ss, endTime=YYYY-MM-DDThh:mm:ss, renewCount=10, delegableCount=10, isSolution=false, type=Saml_Bearer]

  /var/log/vmware/sso/vmware-identity-sts.log (Request Received):

YYYY-MM-DDThh:mm:ss  INFO sts[71:tomcat-http--33] [CorId=UUID] [com.vmware.identity.sts.ws.handlers.SOAPHeadersExtractor] Found 1 {http://docs.oasis-open.org/wss/2004/01/> oasis-200401-wss-wssecurity-secext-1.0.xsd}Security headers
YYYY-MM-DDThh:mm:ss  INFO sts[71:tomcat-http--33] [CorId=UUID] [com.vmware.identity.sts.impl.STSImpl] Entering issue() token...

  /var/log/vmware/sso/vmware-identity-sts.log (Delayed Response of 30~120 seconds later):

YYYY-MM-DDThh:mm[+30~120s]:ss INFO sts[71:tomcat-http--33] [CorId=UUID] [com.vmware.identity.interop.ldap.SslX509EqualityMatchVerificationCallback] Server SSL certificate signature > verified.
YYYY-MM-DDThh:mm[+60~120s]:ss INFO sts[71:tomcat-http--33] [CorId=UUID] [com.vmware.identity.idm.server.IdentityManager] Authentication succeeded for user[<User Name>@<Domain Name>] in tenant [vsphere.local] in [60589] milliseconds with provider [<Domain Name>] of type [com.vmware.identity.idm.server.provider.ldap.LdapWithAdMappingsProvider]

Environment

Aria Automation 8.18.x
vCenter Server 8.x

Cause

This issue is caused by latency within the vCenter Single Sign-On authentication process, which results in the vCenter vAPI service operating in a degraded state.
The resulting timeouts (often taking 60 seconds or more) significantly delay the authentication response. These delays cascade up to the vAPI endpoint, causing Aria Automation's connection requests to time out before vCenter can successfully authenticate the service account.

Resolution

To eliminate the authentication delays affecting the vAPI service, perform one of the following actions:

Option 1: Specify Healthy Domain Controllers: Reconfigure Active Directory over LDAPs Identity Source in vCenter Server. Change the configuration from the "Any Domain Controller" option to explicitly use only the IP addresses or FQDNs of healthy, reachable domain controllers.

Option 2: Fix Network Connectivity: Investigate and resolve the underlying network connectivity, firewall, or routing issues that are preventing the vCenter Server from successfully communicating with the unreachable domain controllers over the LDAPS.