BFD tunnels are down between ESXi hosts in an NSX environment

Products

• VMware NSX

Issue/Introduction

• From within NSX User Interface some host may have a Degraded status.
• From the ESXi Cli, tunnels are in a down state and the sent and recevd values are not actively increasing.

[root@esxi:/etc/init.d] nsxdp-cli bfd sessions list
Remote  Local  local_disc   remote_disc  recvd    sent    local_state local_diag                        client  flaps  bfd_type
<IP>    <IP>   <lo dis>     <re dis>     28454445 2867467 down       1 Control Detection Time Expired  vdl2   1      Tunnel
......

• vmkping between host TEPs is successful.

vmkping ++netstack=vxlan -s 1572 -d -I <source vmk tep>  <destination tep IP>
PING <IP> (<IP>): 1572 data bytes
1580 bytes from <IP>: icmp_seq=0 ttl=64 time=0.103 ms
1580 bytes from <IP>: icmp_seq=1 ttl=64 time=0.096 ms
1580 bytes from  <IP>: icmp_seq=2 ttl=64 time=0.112 ms

• Packet capture on the ESXi may not show any BFD traffic from either TEP. 

pktcap-uw --uplink [VMNIC] --capture UplinkSndKernel,UplinkRcvKernel --rcf 'geneve and port 3784' --ng -o - | tcpdump-uw -enr -

Environment

VMware NSX-T Data Center
VMware NSX

Resolution

Workaround:

Place the ESXi host into Maintenance Mode and restart the nsx-cfgagent

/etc/init.d/nsx-cfgagent restart