Wrong GENEVE inner checksum from NIC firmware on ESXi host with BCM5741x / Broadcom 5741x NIC.

Products

• VMware vSphere ESXi
• VMware NSX

Issue/Introduction

• Between affected VMs pings work
• Connectivity issue with TCP traffic with Geneve overlay, such as when using NSX-T.
• When the VMs are running on two different hosts and the datapath involves the physical adapters, the issue arises.
• Packet captures from within the guest OS of the source VM show that the VM has received the SYN-ACK packet from the target VM. However, no ACK is generated on the source VM.
• "sw encap csum error rx" can be observed in the NIC's private stats of ESXi at the receiving side, assuming these receiving ESXi host also used the Broadcom NIC(bnxtnet):
  [rxq-drss61] sw encap csum error rx: 61964 < -------
[rxq-drss62] sw encap csum error rx: 63750
[rxq-drss63] sw encap csum error rx: 63454
  

 

Note: NIC private stats can be found in /commands/nicinfo.sh.txt in the ESXi support bundle, or by the the following command:

All ESXi host have the script nicinfo.sh available for use.  To find and use this command follow these steps:

1. From root execute the below commands:
   
   • find / -iname nicinfo.sh
     
     output:
     /usr/lib/vmware/vm-support/bin/nicinfo.sh
     
     
2. Run  /usr/lib/vmware/vm-support/bin/nicinfo.sh | less
   
   
   • This is the output of the command piped into the less command.  Piping it to less allows searches to be done in the file easily.
   • The "/" will search for the string down the file from where the cursor is presently located.
   • The "?"  will search up the file from where the cursor is presently located.
   • The format to search down for "sw encap csum error"  would look as follows:
     
     
     
     
   • NOTE:
     • The statistics that are displayed come from the NIC driver.  This means that every vendor presents different statistic their way.  In this specific case the statistic is for a Broadcom NIC.
     • This statistic may exist in other vendor NIC statistic but it may not be labeled the same.  For this article we only care about Broadcom's version of this statistic.  This issue only applies to Broadcom NICs.
3. Another method uses the following command.
   localcli --plugin-dir /usr/lib/vmware/esxcli/int networkinternal nic privstats get -n <vmnicX>

Environment

VMware vSphere ESXi 7.0

VMware NSX

Cause

The issue is caused by the NIC not computing the inner checksum for Geneve traffic when Geneve offload is enabled. The issue was introduced starting with firmware version 223.xx and is also present in versions 224.x & 225.x.

Resolution

The issue is fixed in firmware version 22.61.x and later.
Please be aware that within the VCG a version is listed as 226.1.107.1, this is a differing naming convention and can be considered as 22.61.10.71. Both contain the fix for this issue.

The customer should consult their OEM for the specific version number as it is different depending on the OEM and NIC model. And the matching bnxtnet driver version (according to the VGL or OEM) should be used.

Workaround

• Enable software simulation of Geneve Offload, which will takeover from hardware, using the below command:
  esxcli network nic software set --vmnic=vmnicX --geneveoffload=1

Additional Information

Impact/Risks:
All TCP/IP traffic across the adapters is impacted.