GENEVE tunnels show as down after NSX 4.2.x upgrade in the NSX UI

Products

• VMware NSX

Issue/Introduction

• After a successful upgrade to NSX 4.2.x, one or more NSX prepared clusters are showing TEP (GENEVE) tunnels down. 
• The down TEP tunnels are to the other NSX prepared ESXi hosts and edge nodes. The up TEP tunnels, are to those ESXi hosts in the same cluster. 
• Rebooting impacted ESXi hosts do not resolve the issue.
• Restart of NSX service nsx-proxy, did not resolve the issue.
• Network connectivity between TEP is established and working using TEP to TEP ping.
• VMs on the impacted hosts have connectivity issues.

Environment

VMware NSX 4.2.x

Cause

Some vDS on these hosts did not have their configuration upgraded properly during NSX upgrade, some stale entries are still persisted in the switch config causing the switch update to not be realized. 

Resolution

To workaround this issue, follow the steps below:

1. Enter maintenance mode on the affected hosts and Remove Hosts from a vSphere Distributed Switch.
2. Restart the NSX-proxy service on the hosts using command: /etc/init.d/nsx-proxy restart
3. Allow some time for the NSX manager to sync with the hosts
4. Add the vDS back to the hosts

Alternatively, if above steps did not work:

1. Enter maintenance mode on the affected hosts and Remove Hosts from a vSphere Distributed Switch.
2. Stop the NSX proxy service and remove NSX VIBs from the hosts Uninstall NSX from a vSphere Cluster
3. Wait for NSX VIBs to be removed completely and check: esxcli software vib list | grep -E 'nsx|vsipfwlib'
4. Prepare NSX again on the hosts (installing NSX VIBs)
5. Add the vDS back to the hosts