Addressing Network Packet Loss Due to DRS Workload Concentration in vSphere ESXi

Products

• VMware vCenter Server

Issue/Introduction

In clusters running VMware vSphere ESXi 8.0 Update 3, virtual machines (VMs) are disproportionately distributed across hosts. A large number of VMs may concentrate on specific hosts while peer hosts remain underutilized. This uneven distribution leads to hardware resource exhaustion, and specifically, network packet drops occur on the heavily loaded host due to physical NIC buffer exhaustion.

Environment

• VMware vCenter Server Appliance 8.x
• VMware vSphere ESXi 8.x
  
  

Cause

The Distributed Resource Scheduler (DRS) placement logic evaluates only CPU and Memory utilization when determining optimal VM placement. It does not factor physical network interface (NIC) throughput or I/O contention into its scoring algorithm. Consequently, DRS may consolidate multiple network-intensive VMs on a single host if that host appears to have sufficient CPU and Memory capacity, causing the NIC buffer to saturate and drop packets.

Resolution

To mitigate uneven distribution and associated network packet loss, perform the following actions to optimize DRS placement and isolate high-traffic workloads.

1. Validate DRS Automation Settings
   • Ensure the cluster is set to "Fully Automated" to allow DRS to actively move VMs.
     1. Navigate to the Cluster in the vSphere Client.
     2. Select Configure > vSphere DRS.
     3. Click Edit and verify the Automation Level is set to Fully Automated.
2. Configure Host Power Management
   • To prevent performance degradation during migration and load balancing, configure hosts for high performance to maximize headroom.
     1. Navigate to the Host in the vSphere Client.
     2. Select Configure > Hardware > Power Management.
     3. Set the Policy to High Performance.
3. Review and Adjust Affinity Rules 
   • Restrictive VM-Host affinity rules can prevent DRS from effectively rebalancing workloads across the cluster.
     • Identify active affinity rules within the cluster settings.
     • Disable VM-Host affinity rules temporarily to determine if DRS can successfully rebalance the VMs.
4. Implement Anti-Affinity Rules 
   • For VMs identified as having high network I/O, ensure they are distributed across different physical hosts to reduce per-host buffer pressure.
     
     1. Select the Cluster in the vSphere Client.
     2. Navigate to Configure > Configuration > VM/Host Rules.
     3. Click Add to create a new rule.
     4. Set the rule type to Separate Virtual Machines.
     5. Add the identified high-network-traffic VMs to this rule.
5. Verify NIC Performance 
   • Monitor physical NIC statistics post-implementation to confirm that packet loss has ceased.
     • Use the following ESXi command to view interface statistics:
       esxcli network nic stats get -n vmnic0
     • Observe the Dropped packets counters during peak traffic periods to ensure stabilization.

Additional Information

• Performance Best Practices for VMware vSphere 8.0 Update 3
• VMs Unevenly Distributed by DRS to High-Spec Hosts
• High amount of packet loss observed on physical NIC(s) caused by buffer exhaustion