NSX Management cluster enters a unsupported state with nodes of different versions

Products

• VMware NSX

Issue/Introduction

• You are not in the process of upgrading.
• You are attempting to resize or replace nodes in an existing VMware NSX Manager cluster.
• You manually deploy an NSX Manager node using an OVA version that does not match the existing cluster.
• You execute the join command from the new node's CLI to attach it to the existing cluster.
• The join command completes successfully.
• The NSX Manager cluster enters an unsupported split-version state.

Environment

VMware NSX

VMware NSX-T Data Center

Cause

The NSX CLI join command does not currently perform a strict version compatibility pre-check before attaching an NSX Manager Node to the cluster. This allows an administrator to inadvertently join a Node of a different version, placing the cluster into an unsupported state.

Resolution

This is a known issue impacting VMware NSX. Subscribe to this KB article to be updated if a fix version becomes available. 

 

Workaround

To restore the cluster to a supported state, you must detach the mismatched nodes and replace them with nodes running the correct version. The original product version remains registered in the database, allowing you to revert safely.

Please ensure a backup has been carried out prior to making the below changes.

 

1. SSH into a healthy NSX Manager node running the original, correct version.

2. Run get cluster status to identify the UUIDs of the mismatched nodes.

3. Run detach node <node-uuid> to remove the mismatched nodes from the cluster.

4. Delete the detached VMs from vCenter.

5. If your NSX Managers are connected to a Compute Manager (vCenter):

   1. Log into the NSX UI and navigate to System > Appliances.

   2. Click ADD NSX APPLIANCE. The Auto Deploy function will utilize the correctly registered product version to deploy the replacement nodes.
      
      

6. If your NSX Managers are NOT connected to a Compute Manager:

1. 1. Manually deploy new NSX Manager nodes using the correct version OVA.

   2. On the new nodes, execute the join <Manager-IP> cluster-id <cluster-id> username <Manager-username> password <Manager-password> thumbprint <Manager-thumbprint> command to add them back into the cluster.

If you believe you have encountered this issue, open a support case with Broadcom Support and refer to this KB article.