Intermittent packet drops for dual (VXLAN and Geneve) encapsulated packets

Products

• VMware NSX
• VMware vSphere ESXi

Issue/Introduction

• Intermittent packet drops are observed for double encapsulated packets with VXLAN and Geneve.
  For example, Docker VMs connected to overlay segments on NSX.
  Docker will generate VXLAN encapsulated packets, which will further get encapsulated with NSX Geneve header.
• Checksum offload is enabled on the Guest OS and thus, the VM will set a pseudo checksum value for Outer VXLAN and Inner TCP header, and these checksums along with NSX Geneve header checksum are expected to get offloaded to pNIC/uplink layer on ESXi host.
• In the packet captures taken on receiving host in Wireshark (with Validate the UDP Checksum if Possible enabled in Protocol Preferences), the outer header checksum is computed correctly for the outer UDP headers, however, the inner TCP header is not computed correctly.
• This incorrect checksum results in packet drop at Guest OS
• The issue only occurs with VMXNET3 and not with E1000 adapter.

 

 

Environment

VMware NSX

VMware vSphere ESXi

Cause

The innermost TCP checksum is expected to be computed by vdl2 on ESXi host, which does not support offload for VXLAN packets and thus, vdl2 while adding the NSX Geneve header, will only compute checksum for the outer VXLAN header and not for the innermost TCP header.

Due to this, the Guest OS receives the packet with the pseudo checksum inserted by sender VM, and drops the same due to incorrect checksum.

Resolution

This is a known limitation with vdl2.

Workaround:

Disable checksum offload on Guest OS using either of the below commands:

• ethtool -K eth0 tx-udp_tnl-segmentation off tx-udp_tnl-csum-segmentation off
• /usr/sbin/ethtool -K eth0 tx-checksum-ip-generic off