Packet drops and connection issues due to incorrectly configured load balancing

Products

• VMware vSphere ESXi

Issue/Introduction

The load balancing capability of an ESXi virtual switch, configured in the Teaming and failover settings, offers a number of load balancing options. One of these, Route based on IP hash, requires a matching configuration at the upstream switch connect to the host.

An incorrect load balancing configuration can exhibit the following issues:

• Some remote hosts can be reached from clients connected to the virtual switch, but not all of them
  (i.e. a VMkernel can only ping about half of the IP addresses that are expected to be reachable)
  
  
• Intermittent packet loss
  (i.e. about half of ping responses are lost)

Cause

The Route based on IP hash load balancing algorithm requires static EtherChannel on the uplink switch. If this option was selected but a corresponding configuration is not present on the upstream switch, issues such as packet loss can arise.

Another load balancing feature, Link Aggregation Control Protocol (LACP), requires a matching configuration on the uplink switch to take effect. Furthermore, switch vendors have differing implementations of LACP which can sometimes result in unexpected behavior.

Load balancing across uplinks when the uplinks do not have identical VLAN access, lead to different network segments or topologies, or are split across two or more physical switches where the VPC peering link is down or having issues will also result in network degradation.

The partial unreachability and packet loss behavior described above can also be exhibited when the load balancing is configured correctly but one or more uplinks are degraded, due to a physical connection problem for example.

Resolution

When partial unreachability or intermittent packet loss is observed, 

• Examine the load balancing setting in Teaming and failover. If Route based on IP hash is set, confirm whether static Etherchannel is configured on the switch.
• Try changing the setting to Route based on originating virtual port
• If feasible, move all uplinks except for one to Standby or Unused and individually test each uplink as Active one at a time to identify issues with degraded uplinks
• If LACP is configured on the switch, view the status with the following command
  esxcli network vswitch dvs vmware lacp status get
  A functioning LACP aggregation will show as Bundled. 
• If one or more uplinks are identified as having reachability issues that other uplinks do not: 
  
  • Verify access to VLANs from those uplinks
  • If multiple physical switches are involved, confirm the status of any necessary peering links
  • Reseat the SFP and examine the cable for potential damage
  • Contact your network hardware vendor for further investigation