Error: Unexpected error while upgrading upgrade unit. Failed to Exit node from maintenance mode. retry the operation

Products

• VMware NSX
• VMware NSX-T Data Center

Issue/Introduction

• You are upgrading your NSX-T from 3.2.x  or 4.1.x version to 3.2.x or 4.1.x version
• During the upgrade, host and edge upgrade get's completed successfully
• During the manager upgrade, 1 or more than 1 manager node fails.
• Upgrade status in the NSX-T UI will stuck in "In Progress"
• If you expand the Sequence number 3. Node OS Upgrade in NSX-T UI -> Upgrade tab then you can see that manager node reaches 90% but eventually fails
• In the upgrade details tab, you can see it is throwing "Failed" status with the following error,

"Failed to exit node <Manager UUID> from maintenance mode. Please retry the operation"

• NSX Managers upgrade stopped with Failed status with the error "NSX Managers upgrade has failed, check error details to determine if manual resolution is needed and 'Retry Upgrade'."
• SSH into the manager node which is failing and execute the following command in admin mode,

nsxmgr1> get group maintenance-mode status

Group Type: <name of the service>
Members:
       UUID                           Leadership Work Completed     Group Update Ack Received        Maintenance Mode Status
<Manager 1 UUID>                        True                         False                              MAINTENANCE_MODE_FAILED
<Manager 2 UUID>                        False                        False                              MAINTENANCE_MODE_OFF
<Manager 3 UUID>                        True                         True                               MAINTENANCE_MODE_OFF

Note: Command "get group maintenance-mode status" needs to be entered manually as this command would not auto-complete. 

• All the fields must be True and the Maintenance mode status should be "MAINTENANCE_MODE_OFF" for all three managers.
• If any node is showing the status of "MAINTENANCE_MODE_FAILED" then check the "Group Update Ack Received" that will be in "False" state.

Environment

VMware NSX

VMware NSX-T Data Center

Cause

This issue occurs when Central Control Plane (CCP) messages are not received within the expected timeframe, preventing the node from successfully transitioning out of maintenance mode.

Resolution

Fixed in NSX releases 4.2.2 and 9.0.0 and higher. See Download Broadcom products and software for steps to download this release.

Workaround 1

If an immediate upgrade to a fixed version is not possible, perform the following steps:

1. Log in to all three NSX Manager nodes as root via SSH.
2. Restart the Central Control Plane service:
     /etc/init.d/nsx-ccp restart

3. Wait 10 minutes and verify the status:
     get group maintenance-mode status

4. If the status is still MAINTENANCE_MODE_FAILED, use the following API call to manually disable maintenance mode:
     POST https://<NSX-MGR-IP>/api/v1/cluster-manager/nodes/<NODE-UUID>?action=maintenance_mode_off

5. Resume the upgrade from the NSX UI. Do not use CLI commands to resume the upgrade.

To speak with a customer representative or a Support Engineer, see Contact Support.

Workaround 2

If the manager node is stuck in the reboot then we can see the below log messages in var/log/syslog,

reboot.target: Job reboot.target/start timed out.
<Date>T<Time>Z <hostname> systemd 1 - - Timed out starting Reboot.
<Date>T<Time>Z <hostname> systemd 1 - - reboot.target: Job reboot.target/start failed with result 


In such scenarios, rebooting the manager manually and perform the above workaround 1. 

Additional Information

In case you see the error as "Management Plane node failed to enter maintenance mode" Please refer below KB.
During NSX upgrade the Management Plane node failed to enter maintenance mode