NSX Manager UI and API are inaccessible after reboot/restart when a host-based agent VM service is deployed

Products

• VMware NSX

Issue/Introduction

• The NSX Manager, or the entire NSX Manager cluster were restarted. 
• This may also occur after restart of proton service on NSX Manager. 
• NSX Manager UI login page does not load, or the API is unreachable.
• "systemctl status proton" shows "active (running)", but the UI and API never become reachable.
• Repeated reboots won't rectify the issue.
• This problem is typically seen after an upgrade.
• The symptoms above are usually present together.

Environment

VCF 9.1.1
VMware NSX 9.1.1 and above

Cause

During startup, NSX Manager runs a validation step for host-based agent VM services deployed through VMware ESX Agent Manager (EAM). In some environments this step doesn't complete, and the startup waits indefinitely.

Resolution

This is a known issue impacting VMware NSX. Subscribe to this KB article to be updated if a fix version becomes available.

Workaround:

The workaround needs to be applied to every NSX Manager node in the cluster, one node at a time.

1. SSH to the NSX Manager node as root.
2. Back up the configuration file:
   cp /usr/tanuki/conf/proton-tomcat-wrapper.conf /root/proton-tomcat-wrapper.conf.bak-$(date +%s)
3. Use vi to open /usr/tanuki/conf/proton-tomcat-wrapper.conf for editing:
   vi /usr/tanuki/conf/proton-tomcat-wrapper.conf
4. Find the last "wrapper.java.additional.<N>=" line and note its number (for example, .69).
5. Add these two lines after it, using the next two numbers (for example, .70 and .71):
   wrapper.java.additional.70=-Dsun.net.client.defaultConnectTimeout=20000
wrapper.java.additional.71=-Dsun.net.client.defaultReadTimeout=120000
6. Save the file and restart the service:
   systemctl restart proton
7. Wait for the node to come up.
8. Run "systemctl status proton" and confirm it shows "active (running)" and the UI and API are reachable.
9. Repeat the same steps on the next node until the workaround is applied on all nodes in the cluster.