Aria Suite Lifecycle Manager and Aria Automation fail on Data Collection for Content Library from vCenter Server

Products

• VMware vCenter Server

Issue/Introduction

Data Collection for vCenter endpoint fails on content library section with below error:

2024-11-21T14:07:32.325Z INFO vrlcm[1384] [pool-3-thread-45] [c.v.v.l.d.v.VcCollectionImpl]  -- Errors :: {"type":"com.vmware.vapi.std.errors.internal_server_error","value":{"error_type":"INTERNAL_SERVER_ERROR","messages":[{"args":[],"default_message":"The endpoint, while acting as a gateway, received an invalid response from an upstream server it accessed while attempting to fulfill the request.","id":"com.vmware.vapi.endpoint.badGateway"}]}}
...
2024-11-21T14:07:32.326Z ERROR vrlcm[1384] [pool-3-thread-45] [c.v.v.l.c.v.c.FetchContentLibrariesTask]  -- Exception occures while trying to fetch content library details. Error :
com.vmware.vrealize.lcm.common.vsphere65Plus.exception.UnknownContentLibraryException: Failed to perform operation.
...
2024-11-21T14:07:32.326Z INFO vrlcm[1384] [pool-3-thread-45] [c.v.v.l.p.a.s.Task]  -- Injecting task failure event. Error Code : 'LCMVCCONFIG100004', Retry : 'true',...

 

On vCenter Server vapi-endpoint logs errors relating to a timeout:

2024-12-17T15:34:27.007Z | INFO  | vAPI-I/O dispatcher-0     | SessionApiSecurityUtil         |                                      | Created child session with session manager com.vmware.cis.session on http://localhost:8201/api forsession <USERNAME> (internal id 1axxxxxx).
2024-12-17T15:34:34.324Z | ERROR | vAPI-I/O dispatcher-0     | SessionFacade                  | ba649507-xxxx-xxxx-xxxx-99631850xxxx | Unexpected error occurred while executing the call with session <USERNAME> (internal id 60xxxxxx) for method com.vmware.appliance.vmon.service.list_details.com.vmware.vapi.client.exception.TransportProtocolException: HTTP response with status code 503 (enable debug logging for details): no healthy upstream

...

2024-12-17T15:35:50.692Z | ERROR | vAPI-I/O dispatcher-0     | SessionFacade                  |                                      | Unexpected error occurred while executing the call with session <USERNAME> (internal id 82xxxxxx) for method com.vmware.cis.session.create.com.vmware.vapi.client.exception.ConnectionException: http://localhost:1080/cls invocation failed with "java.net.SocketTimeoutException: 10,000 milliseconds timeout on connection http-outgoing-53168 [ACTIVE]"

Environment

Aria lifecycle manager 8.x
Aria Automation 8.x
vCenter Server 8.0 U3

Cause

Slow Active Directory logins due to AD latency or geographical constraints can impact login performance during data collection, causing the vapi session timeout (10seconds) to be hit upon content library requesting a session.

Resolution

Address AD latency in environment.

 

As a workaround, increase the vapi session timeout value by adding the property:

1. Login to the VCSA commandline interface with root user
2. Backup the endpoint.properties file: cp /etc/vmware-vapi/endpoint.properties /root/endpoint.properties.bck
3. Change the property in the vapi-endpoint properties file: echo "session.child.soTimeout=20000" >> /etc/vmware-vapi/endpoint.properties
4. Restart the service in order for the change to take effect: service-control --restart vapi-endpoint

Additional Information

• The timeout added above is for 20seconds
• Needlessly increasing the timeout to high values can have adverse effects
• Adding the session.child.soTimeout to the endpoint.properties file persists across vCenter upgrades