VMware NSX Manager UI not available and returns a 503 error after upgrade to 4.1.0

Products

• VMware NSX

Issue/Introduction

• An upgrade to VMware NSX 4.1.0 has been completed.
• The UI fails to load and returns a 503 error.
• The cluster is reporting STABLE, confirmed using the "get cluster status" command from the admin CLI on an NSX manager node.
• Checking the logs on the NSX Manager under /var/log/proxy/reverse-proxy.log you see the below errors:

WARN Processing request <uuid-1> MessageOfTheDayUtils 2560 - [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="http"] Failed to get the motd content 
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://null:7441/api/v1/node": null; nested exception is java.net.UnknownHostException: null
...
WARN Processing request <uuid-2> LoginConsentDetails 2566 - [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="http"] Failed to get the consent content
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://127.0.0.1:7440/nsxapi/api/v1/loginbanner": Connection timed out (Connection timed out); nested exception is java.net.ConnectException: Connection timed out (Connection timed out)
...

• In the NSX manager log /var/log/proxy/envoy_access_log we see the below entries:

<ip-aadress-1> <ip-aadress-2> "GET" "/login.jsp" "HTTP/1.1" 503 UAEX 0 0 60001 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36" ",uuid-3>" "<nsx-manager-fqdn>" "-"

NOTE: The preceding log excerpts are only examples. Date, time and environmental variables may vary depending on your environment.

Environment

VMware NSX

Cause

This issue occurs when rendering login.jsp is taking more than 60 seconds and hence we are seeing 503 or timeout issue on login.jsp.

This can be confirmed in the NSX Manager log /var/log/proxy/envoy_access_log when we see the 503 UAEX API result for "GET" "/login.jsp":

<ip-aadress-1> <ip-aadress-2> "GET" "/login.jsp" "HTTP/1.1" 503 UAEX 0 0 60001 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36" ",uuid-3>" "<nsx-manager-fqdn>" "-"

Resolution

This issue is resolved in VMware NSX 4.1.1, available at Broadcom downloads.

If this issue has been experienced, the UI is down after upgrade and the version is NSX 4.1.0 the following workaround can be applied.

1. This procedure is exclusively for NSX version 4.1.0 only.
2. Download the replacement login.jsp file attached to this KB
3. Take the backup of original login.jsp
   cp /opt/vmware/proxy-tomcat/webapps/ROOT/login.jsp /opt/vmware/proxy-tomcat/webapps/ROOT/login.jsp.bkp
4. Apply the new file
   cp /tmp/login.jsp /opt/vmware/proxy-tomcat/webapps/ROOT/login.jsp

Additional Information

Similar issue seen after upgrade to 4.1.0: VMware NSX 4.1.0 UI becomes inaccessible after login due to language setting