TCP session failures over overlay segments due to non-zero padding checksum mismatch

Products

• VMware vSphere ESXi

Issue/Introduction

Application connectivity failures occur for virtual machine workloads due to non-zero padding and checksum mismatch. This issue typically manifests during the TCP three-way handshake, where packets are received but ignored by the destination GuestOS, leading to spurious retransmissions.

The behavior depends on the CSUMVFD flag status:

• Working Scenario: When the CSUMVFD flag is set to 1, the vmxnet3 backend and the guest vNIC driver indicate that the checksum has already been verified for the packet:
  
  
  
  
• Failure Scenario: When the flag is set to 0, the backend and vNIC driver do not mark the packet as verified. Consequently, the guest VM networking stack must perform its own verification. If the guest OS assumes Ethernet padding is always zero according to RFC 1042, or contains bugs when verifying packets with non-zero trailers, it will drop the packet:
  
  

Environment

VMware ESXi 8.0.x
VMware ESX 9.0.x

Cause

The issue arises from a mismatch between how guest OS drivers, hypervisors, and physical network interface cards (NICs) handle non-standard padding during hardware offload operations:

• Non-Zero Padding: Guest operating systems or third-party virtual appliances incorrectly append non-zero "garbage" data instead of zeros to meet the 64-byte Ethernet minimum frame requirement.
• Offload Failure: Physical NIC hardware often fails to calculate or validate checksums correctly when non-zero trailers are present in encapsulated overlay frames (such as GENEVE).
• Flag Mismatch: When the physical NIC fails checksum validation, the hypervisor does not set the CSUMVFD (Checksum Verified) flag. The guest VM stack then attempts its own validation, identifies the padding as invalid according to protocol standards, and discards the packet.

Resolution

This is a known issue impacting VMware ESXi.

Workarounds

1. Disable RX Checksum Offload (Guest level): Force the Guest OS to handle checksumming, bypassing the mismatch:
   • Windows: Disable TCP Checksum Offload (IPv4) in the VMXNET3 adapter properties.
     
     
   • Linux: Run 

     sudo ethtool -K rx off tx off

2. Appliance Configuration: Configure the load balancer or firewall to use zero padding (0x00) only. 

Additional Information

Understanding TCP Checksum Offloading (TCO) in a VMware Environment

TCP sessions fail over GENEVE overlay due to TCP CHECKSUM INCORRECT errors