All host Transport nodes disconnected after power outage in NSX environment

Products

• VMware NSX

Issue/Introduction

For this issue to occur your NSX environment must be set to publish FQDN's instead of using IP addresses. If your datacenter fully goes down and on reboot, of both the host and vm's one of your NSX managers boots before DNS is available it will generate a faulty appliance.xml file since name resolution is not available. This will then get pushed to all of the host in the environment and place the host in a disconnected state.

On the host side in NSX cli you will see the below error, while in the NSX UI the host will be disconnected. Attempting to resolve the issue from the NSX UI will not resolve the issue. Rebooting the host, updating dns, and restarting services will not resolve the issue as well, but you should validate that DNS is up and you can resolve the NSX managers FQDN before you can begin fixing the issue. 

Environment

VMware NSX

Cause

In the event of a complete environmental outage, if the first NSX manager comes up before your DNS server comes up while configured to publish fqdn it will generate an appliance.xml file that does not contain the NSX managers fqdn. This appliance.xml file will then be pushed to all of the host in the environment, even if the other two NSX managers came up after DNS services were restored. As a result all of the host in the environment will be in a disconnected state. 

Resolution

You will first need to identify which NSX manager, has the incorrect appliance.xml file. 

• SSH into all three NSX managers as root 
  • navigate to /etc/vmware/nsx and run the following less appliance.xml
  • one appliance will have the incorrect line: <fqdn>;; communications error to (your dns serverip#53): timed out </fqdn>
  • reboot that appliance so that it will generate a new appliance.xml file
• SSH into a disconnected host as root
  • navigate to /etc/vmware/nsx
  • use vi appliance.xml to edit the xml file.
  • replace <fqdn>;; communications error to (your dns serverip#53): timed out </fqdn> with <fqdn>your.nsxmanager.fqdn</fqdn> 
    • example: <fqdn>nsx-manager-01.lab</fqdn>
     
  • save and exit
  • run the following command /etc/init.d/nsx-proxy restart && /etc/init.d/nsx-opsagent restart to release the old fqdn and query the appliance.xml file once again for the new fqdn.
  • now if you re-enter nsxcli and run get controllers it will show the fqdn of the nsx managers, and will now be in a connected state.
  • copy the edited appliance.xml file to a datastore that is shared across all the host or download it to your local machine.
  • you can now either just copy the appliance.xml file from the datastore or upload it from your local machine to /etc/vmware/nsx and replace the existing file, and then restart the services above
  • validate once again via nsxcli and get controllers
  • Do note that although the get controllers will report a connected state, the UI will take 3-5 minutes to update the host status, but you can validate that you are in a healthy state by performing a vMotion between the first two host that were remediated.