NSX manager shows all tunnels between the hosts Down

Products

• VMware NSX
• VMware NSX-T Data Center

Issue/Introduction

- NSX manager shows all tunnels between the hosts down.
- vmkping between TEPs fail.
- ARP for neighbor TEP is incomplete.
- Packets are getting dropped due to VLAN tag mismatch.
- Software VLAN tagging is disabled. This can be confirmed using 'esxcli network nic software list' on the host.

- In 'pktcap-uw --capture DROP' output we can see drop reason as VlanTag Mismatch.

Captured at Drop point, Drop Reason 'VlanTag Mismatch'. Drop Function 'VSwitchLookup'. TSO not enabled, Checksum not offloaded and verified, Inner Checksum Verified.
SourcePort 22145####, length 136.
Segment[0] ---- 1622 bytes:
0x0000:  0050 56## ##0a 0050 56## ##0c 8100 0997

Environment

VMware NSX
VMware NSX-T Data Center

Cause

In non-working state packet capture on uplink vmnic shows the outer VLAN tag, this indicates physical NIC driver is not doing vlan tagging/untagging properly.

00:50:##:##:##:0a > 00:50:##:##:##:0c, ethertype 802.1Q (0x8100), length 136: vlan ####, p 0, ethertype IPv4, 10.##:##.##.52683 > 10.##:##.##.6081: Geneve, Flags [C], vni 0x12###, proto TEB (0x6558), options [8 bytes]: 00:50:##:##:##:df > 00:50:##:##:##:48, ethertype IPv4 (0x0800), length 74: 172.##:##.##.45346 > 172.##:##.##.8443: Flags [S], seq 48990####, win 64###, options [mss 1460,sackOK,TS val 19300##### ecr 0,nop,wscale 7], length 0


 

Resolution

- Check physical NIC for any issues.
- Check if the NIC firmware/driver is as per compatibility guide.
https://compatibilityguide.broadcom.com/ 

* As a workaround enable software vlan tagging and untagging on ESXi host.

- Validate current software tagging configuration using -
esxcli network nic software list

- Set using following command -
esxcli network nic software set --tagging=1 -n <vmnic#>
esxcli network nic software set --untagging=1 -n <vmnic#>

In the above command replace <vmnic#> with appropriate vmnic ID.

- To unset the configuration use the following command -
esxcli network nic software set --tagging=0 -n <vmnic#>
esxcli network nic software set --untagging=0 -n <vmnic#>