Connectivity Issues with Cilium Custom VXLAN Ports on ESXi 8.0 Update 2

Products

• VMware vSphere ESXi

Issue/Introduction

• Kubernetes pods using Cilium CNI can ping each other across nodes, but TCP/UDP application traffic fails or timeouts.

• Cilium is configured with a custom VXLAN port (e.g., 8223) instead of the standard IANA port 4789.

Environment

VMware vSphere ESXi version: 8.0 Update 2 (Build 23305546).

Cause

The VMXNET3 driver and the virtual switch (vDS) in ESXi 8.0 Update 2 contain a regression/limitation regarding Hardware Offload for encapsulated packets.

When a custom UDP port is used for VXLAN, the ESXi hypervisor fails to recognize the packet as a tunnel. It attempts to perform standard UDP Checksum Offload or Large Receive Offload (LRO) on the outer header incorrectly, or fails to parse the inner header, leading to dropped packets or "Encap Outer Header Errors."

Resolution

The definitive fix is to upgrade the affected hosts to ESXi 8.0 Update 3 (Build 24859861) or later. This version includes:

• Updated VMXNET3 drivers that support dynamic registration of overlay ports.

• Enhanced "Overlay Filters" that allow the NIC to correctly process TEP (Tunnel End Point) traffic for non-standard ports.

Additional Information

Release notes : https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/release-notes/esxi-update-and-patch-release-notes/vsphere-esxi-803-release-notes.html