NSX manager cluster recovery after storage or vSAN failure

Products

• VMware NSX

Issue/Introduction

Following a storage outage or vSAN partitioning event, the NSX Manager cluster may fail to initialize. This typically manifests as an inaccessible UI and services reporting as UNAVAILABLE.

Symptoms

• NSX Manager UI is inaccessible via VIP or Node IP.
• CLI command get cluster status shows services as UNAVAILABLE or UNKNOWN.
• Console displays error: Failed to start default target: Transaction for nsx-custom.target/start is destructive.
• Corfu logs (/var/log/corfu/corfu.9000.log) report DataCorruptionException: Checksum mismatch.

Environment

VMware NSX 4.x

Cause

An underlying storage interruption or improper shutdown causes file system inconsistencies and unrecoverable corruption in the Corfu database binary files.

Resolution

Workaround:

For immediate recovery of a corrupted cluster: 

1. Repair File System: Reboot the affected nodes and run fsck from the GRUB emergency menu to repair partition errors. Follow Resolution section in KB Manager node disk/partition is mounted as read-only alarm in NSX Manager - Disk Corruption correction using FSCK
2. Identify Unrecoverable Nodes: If services fail to start after fsck, run cat /config/corfu/LAYOUT_CURRENT.ds on all nodes. A node with a mismatched or non-replicating layout must be replaced.
3. Detach Faulty Node: On a healthy node, execute the command: detach node <UUID>
4. Redeploy: Delete the corrupted VM from vCenter and deploy a fresh NSX Manager appliance through the NSX UI (System > Appliances).
5. Verify Quorum: Ensure get cluster status reports the cluster as STABLE.
   
   
   Redeploy Scenario Reference: NSX manager node(s) are down in the cluster after issues with storage/vSAN partitioning

Additional Information

NSX manager node(s) are down in the cluster after issues with storage/vSAN partitioning

Manager node disk/partition is mounted as read-only alarm in NSX Manager - Disk Corruption correction using FSCK