Recovering from an NSX Controller failure in NSX for vSphere 6.x

Products

• VMware NSX

Issue/Introduction

This article provides steps to recover from an NSX Controller failure in VMware NSX for vSphere 6.x.

Environment

VMware NSX for vSphere 6.1.x
VMware NSX for vSphere 6.4.x
VMware NSX for vSphere 6.3.x
VMware NSX for vSphere 6.2.x
VMware NSX for vSphere 6.0.x

Resolution

If an NSX Controller fails, you still have two controllers that are working. The cluster majority is maintained, and the control plane continues to function. Despite this, it is important to delete all three controllers and add new ones to maintain a fully functional three-node cluster.

VMware recommends to delete the controller cluster when one or more of the controllers encounter catastrophic, unrecoverable errors or when one or more of the controller virtual machines become inaccessible and cannot be fixed. In such cases, VMware do recommend to delete all controllers, even if some of the controllers seem healthy.

VMware recommends to create a new controller cluster and use the Update Controller State mechanism on the NSX Manager to synchronize the state to the controllers.

To create a new controller cluster and use the update controller state mechanism on the NSX Manager:

1. Log in to vSphere Web Client.
2. Navigate to Networking & Security.
3. Click Installation > Management.
4. In the NSX Controller nodes section, delete all three of them by selecting each one and clicking the Delete Node (x) icon.
   
   Note: When there are no controllers in the system, the hosts operates in "headless" mode. New virtual machines or vMotioned virtual machines will have networking issues until new controllers are deployed and the synchronization is completed.
    
5. Deploy three new NSX Controller nodes by clicking the Add Node (+) icon.
6. In the Add Controller dialog box, select the datacenter on which you are adding the nodes, and configure the controller settings.
    
   1. Select the appropriate cluster.
   2. Select a Host in the cluster and storage.
   3. Select the distributed port-group.
   4. Select the IP pool from which IP addresses are to be assigned to the node.
   5. Click OK.
   6. Wait for installation to complete, and ensure all nodes have a status of Normal.
       
7. Re-synchronize the controller state by clicking Actions > Update Controller State.
   
   Note: Updating the controller state causes VXLAN to be re-synchronized and the logical router instances to be updated.

For more information, see the VMware NSX Administration Guide.

 

Additional Information