Packet loss while using cSwitch and etherchannel (IP HASH and Source MAC Hash)

Products

• VMware vSphere ESXi

Issue/Introduction

This issue occurs when 1 or more of the following conditions are met:

• The system is on ESXi 6.7
• Using DVS 6.6  and using etherchannel
• Standard Switch and etherchannel
• Packet loss is observed when using multiple uplinks to many destination IP addresses 
• pktcap-uw --trace shows packets being dropped after 
  • EtherswitchDispath
  • pktfree  <---- 

Environment

VMware vSphere ESXi 6.7

Cause

There is a problem with the teaming algorithm when queuing multiple packets at the same time. The drop happens when more than 1 packet at the same time attempts to communicate with multiple destinations over multiple network adapters. This mainly effects vSAN heartbeating UDP traffic. This can effect VMs that attempt the same kind of traffic pattern. 

Note: This only effects ESXi 6.7 when using cSwitch teaming policy IP HASH or Source MAC Hash. cSwitch is used by default in 6.7 for standard switches and VDS 6.6

Resolution

This is fixed in ESXi 6.7 U2 in the patch referenced in PR 2238134. Please refer the release notes for ESXi 6.7 U2: Release notes

Workaround:

There are two different workarounds available:

Option1: 

• Remove etherchannel and use teaming policy "Route based on source port". 
• For more information regarding the teaming policy, "Route Based on Originating Virtual Port"

Option 2:

Revert to etherswitch:

• Put host to maintenance
• Enable etherswitch - Disable cswitch
  • esxcli system settings kernel set -s "netVSSUseEtherswitch" -v true
• Reboot host.
• Verify that etherswitch is enabled.
  • esxcli system settings kernel list -o "netVSSUseEtherswitch"

Name                 Type Configured Runtime Default Description
-------------------- ---- ---------- ------- ------- --------------------------------------------------------------------------
netVSSUseEtherswitch Bool true TRUE FALSE Use etherswitch instead of cswitch to back the virtual standard switch.

• Move host out of maintenance

Additional Information

Impact/Risks: Packet loss when using this configuration.