All transport nodes disconnected after upgrade of NSX to 4.2.x

Products

• VMware NSX

Issue/Introduction

In an NSX environment, post-upgrade to NSX 4.2.x, the NSX UI shows all transport nodes as disconnected.

 

Checking the file /etc/vmware/nsx/appliance-info.xml, it shows the following text instead of the FQDN of the NSX Manager: ;; Warning: Client Cookie Mismatch

 

 

Environment

VMware NSX

Cause

This issue is caused by spurious replies to DNS queries, from the DNS server that is in use. 

 

Manually performing reverse lookups from NSX Manager shows a similar text being sent back by the NSX server (perform the DNS lookup a few times as seen below):

root@nsxmgr:~# cd /usr/bin
root@nsxtmgr:/usr/bin# dig -x IP.of.NSX.Manager +short
fqdn.nsxmanager.customer.domain.
root@nsxmgr:/usr/bin# dig -x IP.of.NSX.Manager +short
;; Warning: Client COOKIE mismatch
fqdn.nsxmanager.customer.domain.

 

The presence of the text ";; Warning: Client COOKIE mismatch" above indicates that the DNS server is not replying cleanly and there's an issue with it. It is advisable to resolve this issue with the help of the DNS vendor.

Resolution

It is possible to use a different DNS server using the CLI commands:

nsxmgr> del name-server IP.addr.of.bad.DNS.server

nsxmgr> set name-server IP.addr.of.good.DNS.server

The way NSX Manager handles these spurious responses will be updated in a future NSX release.