NSX Manager upgrade stalled

Products

• VMware NSX

Issue/Introduction

While upgrading the NSX Managers, the progress seems to be stalled and does not move forward for several hours until manual intervention is performed. 

 

The following logline is seen in the file /var/log/proton/nsxapi.log:

Failed to start management application.
org.springframework.context.ApplicationContextException: Failed to start bean 'serviceConfigShardManager'; nested exception is java.lang.IllegalArgumentException: No enum constant com.vmware.nsx.management.upm.model.UpmProfileType.UNRECOGNIZED

Environment

VMware NSX

Cause

When upgrading from an NSX version of 3.2.x or newer, to a newer version, during the Manager upgrade, the progress of the manager upgrade stalls. 

 

This is a result of a software defect that is rare to encounter, where a leader election occurs mid-upgrade.

Resolution

In the vast majority of cases, rebooting the NSX Managers and re-attempting the NSX Manager upgrade will work successfully. However, prior to performing the Manager upgrade again, it's recommended to ensure that a backup of NSX is successfully performed, and if possible also cold clones of all three NSX Manager appliances.

 

If the upgrade fails a second time, it's recommended to collect the data mentioned in Additional Information below, and open a support request with Broadcom Support.

 

Additional Information

If you are contacting Broadcom support about this issue, please provide the following:

• NSX Manager log bundles

• ESXi host log bundles for all hosts where the NSX Manager VMs are running

• Text or screenshots of any error messages seen in NSX GUI or command lines pertinent to the investigation

• The output of the following command run in "root" mode in any one of the NSX Managers: 

  /opt/vmware/bin/corfu_tool_runner.py -n nsx -o showTable --tool corfu-browser -t ServiceConfigPrecedence