API based query to content library using Domain user might fail with "The endpoint,while acting as a gateway,received an invalid response from an upstream server it accessed while attempting to fulfill the request."
search cancel

API based query to content library using Domain user might fail with "The endpoint,while acting as a gateway,received an invalid response from an upstream server it accessed while attempting to fulfill the request."

book

Article ID: 417883

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • API query using an external application to vCenter Server Content Library might return error "The endpoint,while acting as a gateway,received an invalid response from an upstream server it accessed while attempting to fulfill the request."

  • /var/log/vmware/content-library/cls.log: (on vCenter Server)

YYYY-MM-DDTHH:MM:SS | DEBUG    | null             | tomcat-http-32            | AuthenticationFilter           | Selected authentication handler is com.vmware.vapi.cis.authn.SamlTokenAuthnHandler@#######
YYYY-MM-DDTHH:MM:SS | DEBUG    | null             | tomcat-http-32            | OperationMetadataParser        | Param privileges for operation com.vmware.cis.session.create: {}

YYYY-MM-DDTHH:MM:SS | DEBUG    | null             | tomcat-http-32            | AuthorizationFilter            | Validating permissions for 0 objects, in invocation of com.vmware.cis.session.create
YYYY-MM-DDTHH:MM:SS | DEBUG    | null             | tomcat-http-32            | LocalProvider                  | call to invoke() for service 'com.vmware.cis.session', operation 'create'
YYYY-MM-DDTHH:MM:SS | DEBUG    | null             | tomcat-http-32            | InMemorySessionStoreImpl       | Created a new session with id <########-####-####-####-############> for principal Name: '<domain user>', domain: '<domain name>'.

  • /var/log/vmware/vapi/endpoint/endpoint.log: (on vCenter Server)

YYYY-MM-DDTHH:MM:SS | ERROR | vAPI-I/O dispatcher-0     | SessionFacade  | Unexpected error occurred while executing the call with session <user@domain> (internal id #######) for method com.vmware.cis.session.create.
com.vmware.vapi.client.exception.ConnectionException: http://localhost:1080/cls invocation failed with "java.net.SocketTimeoutException: 10,000 milliseconds timeout on connection http-outgoing-24230 [ACTIVE]"
        at com.vmware.vapi.internal.protocol.client.rpc.http.ApacheHttpAsyncClientTransport$FutureCallbackImpl.failed(ApacheHttpAsyncClientTransport.java:252) [vapi-runtime.jar:?]

Caused by: java.net.SocketTimeoutException: 10,000 milliseconds timeout on connection http-outgoing-24230 [ACTIVE]
        ... 11 more
YYYY-MM-DDTHH:MM:SS | WARN  | vAPI-I/O dispatcher-0     | SessionApiSecurityUtil         |                                      | Failed to create child session with session manager com.vmware.cis.session on http://localhost:1080/cls for session <user@domain> (internal id #######).
com.vmware.vapi.endpoint.vapi.ApiException: null

  • /var/log/vmware/envoy/envoy-access.log: (on vCenter Server)

YYYY-MM-DDTHH:MM:SS info envoy[<PID>] [Originator@6876 sub=Default] YYYY-MM-DDTHH:MM:SS GET /api/content/library 500 via_upstream - 0 227 gzip 18009 18009 0 <client machine IP>:3508 HTTP/1.1 TLSv1.2 <VCSA IP>5:443 127.0.0.1:35002 HTTP/1.1 - 127.0.0.1:12346 - -
YYYY-MM-DDTHH:MM:SS info envoy[<PID>] [Originator@6876 sub=Default] YYYY-MM-DDTHH:MM:SS GET /ui/static/resources15180503/ng-next-app/default-libs_common_content-library_src_public_api_ts.js 200 via_upstream - 0 45379 zstd 8 3 4 <client IP>:4102 HTTP/2 TLSv1.2 <VCSA IP>5:443 127.0.0.1:60538 HTTP/1.1 - 127.0.0.1:5090 - -
YYYY-MM-DDTHH:MM:SS info envoy[<PID>] [Originator@6876 sub=Default] YYYY-MM-DDTHH:MM:SS GET /apigw/rest/api/content/library 500 via_upstream - 0 211 zstd 27394 27393 0 <client machine IP>:3558 HTTP/2 TLSv1.2 <VCSA IP>5:443 127.0.0.1:39526 HTTP/1.1 - 127.0.0.1:5090 - -

 

Cause

This issue is caused due to authentication token generation for the user attempting to access the content library is taking more than 10 seconds. VMware vAPI Endpoint service on vCenter server (VCSA) is configured with a default timeout of 10 seconds (10,000 ms).

The time difference between the "Param privileges for operation com.vmware.cis.session.create" and "Created a new session with id <########-####-####-####-############> for principal Name:" in the content library logs will be more than 10 seconds.

Resolution

In order to resolve the issue, proceed to adjust the timeout for "session.child.soTimeout"

  • Log in to VCSA via ssh using root credentials
  • Take a backup copy of the "endpoint.properties" file

cp /etc/vmware-vapi/endpoint.properties /root/

  • Add the timeout parameter

echo "session.child.soTimeout=25000" >> /etc/vmware-vapi/endpoint.properties

Note: The above value implies a timeout of 25 seconds

 

  • Restart the VMware vAPI Endpoint service

service-control --restart vmware-vapi-endpoint

Powered by Wolken Software