VMware NSX Manager responds with 'HTTP ERROR 503' in web browsers and to API calls

Products

• VMware NSX

Issue/Introduction

• Navigating to the NSX Manager UI in web browsers results in a 'HTTP ERROR 503' before hitting the log-in page. 

• The same NSX Manager responds to REST API commands also with '503' errors. 
• NSX Manager envoy_access_log shows '503 UAEX' errors. 

[2024-11-01T15:41:28.518Z] 10.200.8.3 10.200.8.19 "GET" "/api/v1/node/version" "HTTP/1.1" 503 UAEX 0 0 59997 - "10.200.8.3" "vAPI/2.52.0 Java/17.0.10 (Linux; 5.10.214-1.ph4; amd64)" "6db5cf7e-####-####-####-9226406b2876" "a#######01nsx01.#####.com:443" "-"
[2024-11-01T15:42:12.918Z] 10.200.3.210 10.200.8.19 "GET" "/api/v1/node" "HTTP/1.1" 503 UAEX 0 0 59996 - "10.200.3.210" "" "db640706-####-####-####-26c47e611fd1" "a#######01nsx01.#####.com" "-"

• • In the logs above, the final entry on the line is where this request was forwarded.  Here we see "-" because the request was not forwarded.  
  • The status code UAEX indicates that envoy's call to ext_authz (in this case auth server) was responsible for the result.

Environment

VMware NSX 4.x

Cause

Calls from NSX Manager 'auth' service to vIDM hangs until auth service can no longer respond.  

In versions NSX 4.0+ the reverse proxy consists of the envoy proxy that performs the TLS termination on port 443 as well as handling the upstream forwarding and relaying the response. The other component is the auth server, a JVM (java) that performs authentication (authN) to determine whether or not each request should be allowed to pass upstream (and session management, login handling, etc.).  Envoy waits 60 seconds for auth server to respond. If no response is received then the request is rejected with 503, as seen in this scenario.

Resolution

Fix:

• Fixed in VMware NSX 4.2.1
  • A timeout was added for established sessions from the auth server to vIDM to help prevent possible 'auth' service out of memory or thread exhaustion problems related to vIDM external authentication sessions. 

Workaround:

• Restart the affected NSX Manager 'auth' service. 
  
  • From admin user SSH shell; 

restart service auth

• Or reboot the affected NSX Manager.