NSX Upgrade error on Edge nodes: Failed to verify bundle: Bundle version is not compatible with node version

Products

• VMware NSX

Issue/Introduction

• While trying to upgrade NSX to a specific target version, all the pre-checks have passed but the edge upgrade fails at 1% with below error
  
  Prepare edge upgrade bundle http://##.##.##.##:8080/repository/<target NSX build number>/Edge/nub/VMware-NSX-edge-<target NSX build number>.nub failed on edge TransportNode 59####c-a##9-1###-b###-0####6b###ce: clientType EDGE , target edge fabric node id 59####c-a##9-1###-b###-0####6b###ce, return status Download and verify bundle failed with msg: Checking upgrade bundle /var/vmware/nsx/file-store/VMware-NSX-edge-<target NSX build number>.nub contents Verifying bundle VMware-NSX-edge-<target NSX build number>.bundle with signature VMware-NSX-edge-<target NSX build number>.bundle.sig Failed to verify bundle: Bundle version is not compatible with node version. Please refer upgrade compatibility matrix document - https://www.vmware.com/resources/compatibility/sim/interop_matrix.php#upgrade
  
  
• There are NSX edges at higher version than the managers (possibly due to a manager backup restore operation performed leaving managers at older version than the edges)
  
  Below is a sample screenshot showing edges and managers at different versions:
  
  
  
  
• The target mub file uploaded is not compatible with the existing edge version (which is at higher version than managers)

Environment

VMware NSX
VMware NSX-T DataCenter

Cause

Upgrade coordinator looks at the overall version of the setup which is stored in /repository/current-version file on the UA when doing the compatibility check and since current version on managers is compatible with the target version, the pre checks succeeded.
However, edges being at higher version is incompatible with the target version and hence we see edge upgrade fail with error stating incompatible nub version.

Resolution

Workaround:

Upload a new mub file with compatible target version with the current edge version by using upload MUB file option and restart the upgrade for the new target version.