Packet loss while using cSwitch and etherchannel (IP HASH and Source MAC Hash)

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:
On ESXi hosts where:

• The system is on ESXi 6.7
• Using DVS 6.6  and using etherchannel
• Standard Switch and etherchannel
• Packet loss is observed when using multiple uplinks to many destination IP addresses 
• pktcap-uw --trace shows packets being dropped after 
  • EtherswitchDispath
  • pktfree  <---- 

Environment

VMware vSphere ESXi 6.7

Cause

There is a problem with the teaming algorithm when queuing multiple packets at the same time. The drop happens when more than 1 packet at the same time attempts to communicate with multiple destinations over multiple network adapters. This mainly effects vSAN heartbeating UDP traffic. This can effect VMs that attempt the same kind of traffic pattern. 

Note: This only effects ESXi 6.7 when using cSwitch teaming policy IP HASH or Source MAC Hash. cSwitch is used by default in 6.7 for standard switches and VDS 6.6

Resolution

Fix in 6.7 U2 or the patch referenced in PR 2221818 




Workaround:

There are two different workarounds available:

a) Remove etherchannel and use teaming policy "route based on source port"

b) Revert to etherswitch:

 

1. Put host to maintenance

 

2. Enable etherswitch - Disable cswitch

esxcli system settings kernel set -s "netVSSUseEtherswitch" -v true

Reboot host

 

3. Verify that etherswitch is enabled

esxcli system settings kernel list -o "netVSSUseEtherswitch"
Name Type Configured Runtime Default Description
-------------------- ---- ---------- ------- ------- --------------------------------------------------------------------------
netVSSUseEtherswitch Bool true TRUE FALSE Use etherswitch instead of cswitch to back the virtual standard switch.

 

4. Move host out of maintenance

Note: This issue should be resolved in 6.7 U2. Please see PR 2221818

Additional Information

Impact/Risks:
Packet loss when using this configuration