VMware NSX upgrade fails to place manager node in maintenance mode

Products

• VMware NSX

Issue/Introduction

• VMware NSX/VMware NSX-T Data Center is being upgraded from a VMware NSX 4.x version below 4.1.2
• The upgrade target version is NSX 4.1.2 or higher.
• At the Manager upgrade step, the upgrade fails when one of the NSX manager nodes fails to enter maintenance mode.
  
• Clicking the VIEW link displays the error details stating: "Unexpected error while upgrade upgrade unit: Failed to put node <name> in maintenance mode. Please retry the operation after checking 'get group maintenance-mode'..."
  
• You see "MAINTENANCE_MODE_FAILED" on the NSX manager as admin when you run the following command:
  > get group maintenance-mode status
• Results (below is a snipped of the result):
  Group Type: MANAGER
Members:
    UUID                                  Leadership Work Completed   Group Update Ack Received   Maintenance Mode Status
    ########-65d4-4762-97af-############  False                       True                        MAINTENANCE_MODE_FAILED
    ########-c644-d0ab-6369-############  True                        True                        MAINTENANCE_MODE_OFF
    ########-2fdf-4fbd-bcc0-############  True                        True                        MAINTENANCE_MODE_OFF

Environment

VMware NSX-T Data Center 3.x
VMware NSX 4.x

Cause

As of VMware NSX 4.x, some services under the MANAGER (proton) group, such as VMC_* are no longer in use, however stale leases for those deprecated services (from older versions like 3.x) can still remain undeleted after upgrading to NSX 4.x. The presence of those stale leases prevent an NSX Manager appliance from entering into maintenance mode which causes an upgrade of the node to fail at that step. 

Resolution

This issue is resolved in VMware NSX 4.1.2 available at Broadcom Downloads.
If you are having difficulty finding and downloading software, please review the KB article Download Broadcom products and software.

Note the fix applies to upgrades done from version 4.1.2 to a higher version.

If you believe you have encountered this issue and you require a workaround, please open a support request with Broadcom Support and refer to this KB article.