NSX manager node is down and unrestorable

Products

• VMware NSX

Issue/Introduction

• The NSX manager node is down and cannot be restored using standard service restarts. The node is completely unresponsive, lacking SSH access, and displaying a black screen in the console.
  
  
• When accessing the NSX manager console, the node fails to boot and displays errors such as Failed to start default target or UNEXPECTED INCONSISTENCY.
  
  
  
  
• Further verification from a healthy manager node via SSH confirms the issue:

  • Running get cluster status shows the faulty node as DOWN or UNAVAILABLE in the DATASTORE group.

  • Switching to root (st en) and checking the layout file (cat /config/corfu/LAYOUT_CURRENT.ds) reveals the faulty node's IP address listed in the unresponsiveServers section.
    
    

Environment

VMware NSX

Cause

• This issue is caused by unrecoverable file system corruption or a failure of the Corfu datastore cluster on the affected node.
• Standard service restarts and reboots do not clear the node from the unresponsiveServers list.
• Additionally, manual fsck repair attempts on the corrupted file system fail, confirming the node is in an unrestorable state.

Resolution

Because the file system corruption and Corfu datastore failure cannot be repaired manually or via standard restarts, the node must be entirely removed from the cluster database.

To resolve this issue, detach the unrestorable node from the cluster database and deploy a fresh appliance:

1. Establish an SSH session to a healthy manager node.

2. From the CLI of the healthy node, detach the faulty node by running the following command:

   detach node <UUID-of-faulty-node>
   

3. In vCenter, power off and delete the faulty NSX manager virtual machine.

4. Log into the NSX user interface and navigate to System > Appliances.

5. Click the Add Appliance button to deploy a replacement node.

Additional Information

Replacing a faulty NSX-T manager node in a VCF environment