The geneve tunnel is flapping after upgrade NSX

Products

• VMware NSX

Issue/Introduction

• NSX upgraded from 3.x to 4.x
• Packet lost intermittently when ping between two VTEPs
• In ESXi /var/run/log/vmkernel.log shows BFD is flapping:
  
[nsx@6876 comp="nsx-esx" subcomp="bfd"]local: <local-ip-address>, remote: <remote-ip-address>, oldState: down, newState: up, diag: No Diagnostic, type: overlay
[nsx@6876 comp="nsx-esx" subcomp="bfd"]local: <local-ip-address>, remote: <remote-ip-address>, oldState: up, newState: down, diag: Neighbor Signaled Session Down, type: overlay


• In ESXi /var/run/log/vmkernel.log shows vmnic link flapping which is used for the uplink of NSX DVS:
  
netschedHClk: NetSchedHClkNotify:4654: vmnic4: link up notification
netschedHClk: NetSchedHClkWatchdogSysWorld:6522: vmnic4: link up event received, device running at 10000 Mbps so setting queue depth to 86460 bytes with expected 1310 bytes/us
Ipfix: IpfixPortDisableWithLock:1265: Disable non-uplink port. Uplink count: 2. Non-uplink port count: 1. Not need uplink update.
netschedHClk: NetSchedHClkNotify:4654: vmnic4: link up notification
netschedHClk: NetSchedHClkWatchdogSysWorld:6522: vmnic4: link up event received, device running at 10000 Mbps so setting queue depth to 86460 bytes with expected 1310 bytes/us
netschedHClk: NetSchedHClkNotify:4654: vmnic4: link up notification

Environment

VMware NSX 4.x
VMware NSX 3.x

Cause

This can be caused by unplugging the network cable or administratively downing the physical switchport. If this was not an intended link outage it will likely be an issue with the driver, firmware, SFP+ module, cable, and/or switchport of the physical switch. 

Resolution

1. Work with hardware vendor or physical network team to validate the link/switchport to isolate and fix the flapping issue.
2. Check the compatibility between driver and firmware of network adapter. More information can be read from Determining Network/Storage firmware and driver version in ESXi
   Determining Network/Storage firmware and driver version in ESXi