NSX Manager upgrade fails or pre-upgrade checks cannot run due to stale cluster membership entries left by a removed node

Products

• VMware NSX

Issue/Introduction

When upgrading an NSX Manager cluster, the upgrade fails or the pre-upgrade checks cannot complete because the upgrade coordinator is still aware of a manager node that is no longer part of the cluster.

One or more of the following symptoms may be present (not all are required):

• Pre-upgrade checks fail to start with:

             "Cannot execute upgrade checks. Upgrade Coordinator is unreachable on Manager node <node-name>-<ip-address>. Please ensure all Manager nodes are reachable and NSX services are running."

The node named in the error is one that was previously removed from the cluster.

• A pre-upgrade check fails with:

             "Repository Sync status is not success on node <ip-address>" where the IP address does not belong to any current cluster member.

• GET /api/v1/cluster/nodes returns more Manager nodes than GET /api/v1/cluster, with the extra entry corresponding to a node that was removed.
  
  
• The NSX UI shows a Manager node that no longer exists in the cluster.

• On NSX 9.2.0.0 and later, the pre-upgrade data consistency check reports:

                "Found N stale GMLE entries (N LeadershipLease, N GroupMaintenanceMode member(s), N ClusterNodeConfigModel) not present in the cluster configuration"

Environment

VMware NSX 4.1.x, 4.2.x, 9.0.x, 9.1.x

Cause

This typically occurs after a Manager node has been removed, replaced, or redeployed, or after a node removal that did not complete successfully.

Resolution

When a Manager node is removed from the cluster, some of its cluster membership records can remain in the NSX clustering configuration tables. This can happen when a node removal does not complete successfully, or when a node is removed while it is unreachable.

 

The upgrade coordinator determines the set of Manager nodes to upgrade from these stored cluster node records rather than from live cluster membership. A leftover record is therefore treated as a real Manager node, and the upgrade fails when it attempts to contact or verify a node that no longer exists.

Because the stale record is only evaluated during an upgrade, the cluster can appear healthy in normal operation and the problem surfaces only when an upgrade is attempted.