Wrong GENEVE inner checksum from NIC firmware on ESXi host with BCM5741x NIC.

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Between affected VMs the ping work.
• Connectivity issue with TCP traffic with Geneve overlay, such as when using NSX-T.
• When the VMs are running on two different hosts and the datapath involves the physical adapters, the issue arises.
• Packet captures from within the guest OS of the source VM show that the VM has received the Syn-Ack packet from the target VM. However, no ACK is generated on the source VM.
• "sw encap csum error rx" can be observed in the NIC's private stats of ESXi at the receiving side, assuming these receiving esxi host also used the Broadcom NIC(bnxtnet):
        [rxq-drss61] sw encap csum error rx: 61964 < -------
        [rxq-drss62] sw encap csum error rx: 63750
        [rxq-drss63] sw encap csum error rx: 63454

Note: NIC private stats can be found in nicinfo.sh.txt in the ESXi support bundle, or by the the following command:

localcli --plugin-dir /usr/lib/vmware/esxcli/int networkinternal nic privstats get -n <vmnicX>

Environment

VMware vSphere ESXi 7.0

Cause

The issue is caused by the NIC not computing the inner checksum for Geneve traffic when Geneve offload is enabled. The issue was introduced starting with firmware version 223.xx and is also present in versions 224.x & 225.x.

Resolution

The issue is fixed in firmware version 226.1.x and later. The customer should consult their OEM for the specific version number as it is different depending on the OEM and NIC model. And the matching bnxtnet driver version (according to the VGL or OEM) should be used.

Workaround:
Enable software simulation of Geneve Offload, which will takeover from hardware.
The command is: 

esxcli network nic software set --vmnic=vmnicX --geneveoffload=1

Additional Information

Impact/Risks:
All TCP/IP traffic across the adapters is impacted.