Some VM TCP sessions don't work correctly

Products

• VMware vSphere ESXi

Issue/Introduction

• TCP sessions to a specific machine do not complete
• You are using Generic Network Virtualization Encapsulation overlay between client and server machine
• A client machine sends the SYN packet, The server machine replies with SYN/ACK, The client machine keeps sending the SYN packet however it does receive the SYN/ACK
• The packet with the SYN/ACK displays TCP CHECKSUM INCORRECT
   ###.###.###.###    ###.###.###.###    TCP    122    4380    9100 → 54465 [SYN, ACK] Seq=0 Ack=1 Win=4380 [TCP CHECKSUM INCORRECT] Len=0 MSS=1460
• All packets with checksum error ("Checksum: incorrect") have non-zero trailers.  
• The encapsulated packet shows a non zero trailer (7172)

 Generic Network Virtualization Encapsulation, VNI: 0x#####
 Ethernet II, Src: ######## (######), Dst: ####### (######)
    Destination: ###### (########)
 Source: <guest OS server MAC>
    Type: IPv4 (0x0800)
    [Stream index: 1]
    Trailer: 7172 
 Internet Protocol Version 4, Src: #.#.#.#, Dst: #.#.#.#
Transmission Control Protocol, Src Port: 9100, Dst Port: 55340, Seq: 0, Ack: 1, Len: 0

Environment

VMware vSphere ESXi 7.0

Cause

• The server VM guest OS is padding with non zero, "7172" in this case as seen on packet trailer
• The inner TCP checksum includes the trailer for the calculation. 
• Non-zero padding/trailer would lead to incorrect checksum value by NIC checksum offload. 

Resolution

Set checksum offload in software for the nic sending the traffic.

#esxcli network nic software set --ipv4cso=1 -n vmnicX

 

 

Additional Information

esxcli network Commands