SDDC Manager upgrade of NSX Managers fails with "NSX Upgrade Coordinator - Failed to upload NSX bundle. Review the LCM log files on the SDDC Manager appliance"

Products

• VMware NSX
• VMware SDDC Manager

Issue/Introduction

• SDDC Manager initiated an upgrade of NSX Managers from 9.0.0 to 9.0.2
• SDDC Manager GUI shows the following message:
  
  NSX Upgrade Coordinator - Failed to upload NSX bundle. Review the LCM log files on the SDDC Manager appliance at 127.0.0.1:/var/log/vmware/vcf/lcm and resolve identified errors. If resolved, then run the upgrade prechecks and retry the upgrade. If you are unable to resolve the errors, contact Broadcom Technical Support for assistance.


• NSX Managers have been up for an extended period of time, more than 23 days

Environment

VCF 9.0.0

SDDC Manager 9.0.0 

NSX 9.0.0

Cause

NSX 9.0.0 is susceptible to a known issue in JDK-8330017 as described in NSX is Impacted by JDK-8330017: ForkJoinPool Stops Executing Tasks Due to ctl Field Release Count (RC) Overflow. If the managers have been up for an extended period of time, this condition can affect/prevent the upgrade.

Resolution

Perform a rolling reboot of the NSX Managers as instructed in NSX is Impacted by JDK-8330017: ForkJoinPool Stops Executing Tasks Due to ctl Field Release Count (RC) Overflow:

1. Reboot the first NSX Manager.
2. SSH to a Manager as admin user and check cluster health: get cluster status
3. When all services report up on all 3 NSX Manager nodes, reboot the next Manager.
4. Repeat steps 2-3 for the third Manager.