ESXi Host Transport Node Tunnels Display Down Status in NSX UI

Products

• VMware NSX

Issue/Introduction

• NSX UI Reports Down Tunnels status for ESXi Transport Nodes.

• vmkping between the hosts TEP works fine:

vmkping -S vxlan -s 1460 -I vmkX -c 100 <Remote-Host-TEP-IP>

 

• nsxdp-cli bfd sessions list command on host shows all the tunnels are in "init" state.

 Remote              Local             local_disc          remote_disc         recvd               sent                local_state         local_diag          client              flaps               bfd_type

<Remote-ip>      <Local-ip>             a096b2e7            a9edd94f            7692069             7345673             init                No Diagnostic       vdl2                0                   Tunnel

<Remote-ip>       <Local-ip>            21a7e4db            60ab315c            7346101             9760540             init                No Diagnostic       vdl2                0                   Tunnel

 

• Packet Captures on respective TEP (vmk10 or vmk11) interfaces and associated uplink (vmnicX):

1.vmk11: BFD traffic is egressing from the TEP vmk interface.

#pktcap-uw --vmk vmk11 --dir 2 -o - | tcpdump-uw -enr - | grep -i <remote TEP IP> | grep -i <Local TEP IP> 

2.vmnicX: BFD traffic is not egressing from the ESXI Host uplink.

#pktcap-uw --uplink vmnicX --capture UplinkSndKernel -o - | tcpdump-uw -enr - | grep -i <remote TEP IP> | grep -i <Local TEP IP>

 

3. Running Packet trace shows drop at esxi firewall.

#pktcap-uw --trace --srcip <Local TEP IP> --dstip <remote TEP IP>

09:20:57.693566[7] Captured at PktFree point, Drop Reason 'Firewall Drop'. Drop Function 'DVFilterInputOutputIOChainCB'. TSO not enabled, Checksum not offloaded and not verified, SourcePort <port-ID> , length 66.

        PATH:

          +- [09:20:57.693543] |                        PortInput |   <port-ID> |

          +- [09:20:57.693544] |                          IOChain |            | [email protected]#v2_9_0_0

          +- [09:20:57.693545] |                      PreDVFilter |            |

          +- [09:20:57.693560] |                             Drop |            |

          +- [09:20:57.693564] |                          PktFree |            |

 

• To verify the bfdDP rule is disable on the ESXi Firewall.

esxcli network firewall ruleset list | grep -i bfd

Name                           Enabled  Enable/Disable configurable  Allowed IP configurable
-----------------------------  -------  ---------------------------  -----------------------
bfdDP                            false          true                     true

Environment

VMware NSX

Cause

ESXi host's firewall is dropping outgoing BFD traffic (UDP 3784) because the corresponding bfdDP rule is disabled.

Resolution

• Enable the bfdDP rule on the ESXi host firewall. This will allow outgoing BFD traffic and ensure the tunnels report their correct status.

ESXI--->Configure--->System--->Firewall--->Outgoing---click on Edit (search bfdDP in quick filter and check box the bfdDp rule)