NSXT upgrade failed in precheck stage with error "NSX Controller cluster is unstable"

Products

• VMware NSX

Issue/Introduction

• The NSX-T upgrade failed during the precheck stage with the error: 'NSX controller cluster is unstable. Check for network issues or look in the individual nsx controller logs
for failure'.
  
• API output of -  https://<nsx-manager-ip>/api/v1/cluster/status shows  control_cluster_status is DEGRADED/UNKNOWN

Environment

VMware NSX

Cause

The cause of this issue is that the Management Plane is not stable when the Deactivate Cluster operation is initiated, this results in stale entries in the Local NSX Managers.

Resolution

1.  Collect and inspect the API output for stale NSX Manager nodes
   GET https://<nsx-manager-ip>/api/v1/cluster/nodes/deployments
   (From the command line - curl -k -u admin -H "Content-Type: application/json" -X GET https://localhost/api/v1/cluster/nodes/deployments )
2. Collect output of API command
   GET https://<nsx-manager-ip>/api/v1/cluster-manager/status  
   (From the command line - curl -k -u admin -H "Content-Type: application/json" -X GET https://localhost/api/v1/cluster-manager/status )
3. Compare the output of the above API commands to find the stale node ID.
4. Delete the stale node ID using following command 
   DELETE https://<nsx-manager-ip>/api/v1/cluster/nodes/<node-UUID>
   (From the command line - curl -k -u admin -H "Content-Type: application/json" -X DELETE https://localhost/api/v1/cluster/nodes/<node-UUID>)