VMware NSX Manager responds with 'HTTP ERROR 503' in web browsers and to API calls

Products

• VMware NSX

Issue/Introduction

• Navigating to the NSX Manager UI in a web browser, results in a 'This page isn't working' result with error 'HTTP ERROR 503' before getting to the login page:

• The same NSX Manager responds to REST API commands also with HTTP response code 503. 
• In the NSX Manager log /var/log/proxy/envoy_access_log HTTP response code 503 with UAEX can be seen for these API calls: 

10.1.1.3 10.1.1.9 "GET" "/api/v1/node/version" "HTTP/1.1" 503 UAEX 0 0 59997 - "10.1.1.3" "vAPI/2.52.0 Java/17.0.10 (Linux; 5.10.214-1.ph4; amd64)" "6db5cf7e-####-####-####-9226406b2876" "a#######01nsx01.#####.com:443" "-"
10.1.1.2 10.1.1.9 "GET" "/api/v1/node" "HTTP/1.1" 503 UAEX 0 0 59996 - "10.1.1.2" "" "db640706-####-####-####-26c47e611fd1" "a#######01nsx01.#####.com" "-"

Note: In the logs above, the final entry on the line is where this request was forwarded. Here we see "-" because the request was not forwarded. The status code UAEX indicates that envoy's call to ext_authz (in this case authentication server) was responsible for the result.

Environment

VMware NSX 4.x

Cause

Calls from NSX Manager 'auth' service to vIDM hangs until auth service can no longer respond.  

In versions NSX 4.0+ the reverse proxy consists of the envoy proxy that performs the TLS termination on port 443 as well as handling the upstream forwarding and relaying the response. The other component is the auth server, a JVM (java) that performs authentication (authN) to determine whether or not each request should be allowed to pass upstream (and session management, login handling, etc.).  Envoy waits 60 seconds for auth server to respond. If no response is received then the request is rejected with 503, as seen in this scenario.

Resolution

This issue is resolved in VMware NSX 4.2.1, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

A timeout was added for established sessions from the auth server to vIDM, to help prevent possible 'auth' service out of memory or thread exhaustion issues related to vIDM external authentication sessions. 

 

To workaround the issue, you can perform either A or B below:

A. Restart the affected NSX Manager 'auth' service, as admin user, run:

restart service auth

B. Reboot the affected NSX Manager.