Recovering an NSX Manager cluster after 1 node has been powered off for a period of time

Products

• VMware NSX

Issue/Introduction

An NSX Manager node that has been isolated or powered off for an extended period (e.g., several days) needs reintroduction to a functional two-node cluster. There is a concern regarding potential database corruption or cluster instability when the isolated node is powered back on.

• One NSX Manager node (e.g., node1) has been powered off or network-isolated for more than 48 hours.
• The remaining two nodes (e.g., node2, node3) are functional and forming a stable cluster.
• get cluster status on healthy nodes shows the isolated node as DOWN or UNAVAILABLE in the DATASTORE group.

Environment

VMware NSX

Cause

The NSX Manager uses a Corfu distributed database. When a node is isolated, it falls behind in the database epoch. The cluster maintains quorum with the remaining two nodes, but the isolated node lacks recent state updates.

Resolution

The NSX Manager cluster architecture is designed to handle the reintroduction of isolated nodes through automatic synchronization. Follow these steps to recover the three-node cluster:

1. Perform a backup of the current functional NSX Manager cluster.
   • For details on creating backups, refer to How to Configure and Manage VMware NSX Manager Backups.
2. Power on the isolated NSX Manager node.
3. Monitor the synchronization process from the CLI of the powered-on node:

   get cluster status
   

4. Verify that the node status transitions from STARTING or DEGRADED to UP as it syncs with the active nodes.
5. If the node fails to sync or reports DataCorruptionException in /var/log/corfu/corfu.9000.log, the node must be detached and redeployed.

For further assistance, Contact Broadcom Support or see Support Phone Numbers.