VMware NSX Upgrade fails in NSX Edges Upgrade step

Products

• VMware NSX

Issue/Introduction

• You're currently upgrading VMware NSX.
• During the edge precheck, it may fail or stop or the upgrade progress stops and it eventually fails.
• All Edges that are being upgraded report failed state, this may be at different upgrade stages on each edge node.
• The output of the below command (run as root user on the NSX-T manager which is orchestrator node), indicates the number of threads are larger than 4500 (4500 is the threshold for Managers with Large Size Deployment):

sudo -u uuc jstack <UC pid> | grep "tid=" | grep "rpc" | wc -l

Note: <UC pid> is the process of the uuc JVM. It can be identified using the below command as root user:
§ ps aux | grep uuc

Environment

VMware NSX-T Data Center

Cause

The upgrade coordinator service creates a ThreadPoolExecutor for each RPC request.
The code neglects to close the ThreadPoolExecutors after the completion of the RPC request. Consequently, ThreadPoolExecutors accumulate over time causing the upgrade to fail when it passes the threshold.

Resolution

This issue is resolved in VMware NSX 4.1.1, available at Broadcom Downloads.
If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

To workaround this issue, please restart the install-upgrade service on the orchestrator node.

• To identify the orchestrator node, from any NSX Manager admin shell, execute "get service install-upgrade" and verify the "Enabled on" manager node.
  
• From the NSX Manager admin shell on the identified orchestrator node, running the below command:

restart service install-upgrade

• If the above provided an error, you can manually stop the install-upgrade service on the NSX Manager orchestrator node, by executing the below commands from root shell:

/etc/init.d/install-upgrade stop

/etc/init.d/upgrade-coordinator stop
/etc/init.d/lighttpd stop

Then kill the Process IDs:
ps aux | grep uuc
kill -9 <pid>
Note: Normally there are 2 PIDs for the UUC that needs to be killed.

• Then start the install-upgrade service, by running the below command from the admin shell:

start service install-upgrade
 

If starting the service failed from the admin shell, then proceed with the below commands from the NSX Manager root shell:
/etc/init.d/install-upgrade restart
/etc/init.d/upgrade-coordinator restart
/etc/init.d/lighttpd restart