Observing high guest OS storage latency (w_await) and degraded performance during heavy application transactions

Products

• VMware vSphere ESXi
• VMware vSAN
• VMware Telco Cloud Platform

Issue/Introduction

• Guest OS metrics (iostat) record severe latency with  high w_await in Linux or Disk sec during high I/O operations which significantly degraded application performance.
• vSAN application-level write latency exceeding 1,000 ms.
• vSAN Skyline Health Performance graphs (Monitoring > vSAN > Performance > Physical Adapter) show high pNIC vSwitch Port Drop Rate for some of the hosts.
• The issue is observed on VMs hosting applications such as database that handle large number of transactions.
  
  
  
  
• ESXi hosts reporting Receive Missed Errors or Rx Length Errors on physical vmnics.
  
  

Environment

• vSphere 7.0 U3
• vSAN 7.0 U3
• TCP 2.7

 

Cause

• The "pNIC vSwitch Port Drop Rate" indicates that network packets are being discarded at the internal boundary between the hypervisor's virtual switch (vSphere Standard Switch or vSphere Distributed Switch) and the host's physical network adapter (pNIC/vmnic).
• This typically occurs due to buffer exhaustion during sudden traffic microburst or massive surges of data that occur in fractions of a second.
  
  
• Outbound Drops (Tx):
  
  
  • The virtual machines are generating data faster than the physical network adapter can transmit it onto the physical wire.
  • This happens when the pNIC's Transmit (Tx) ring buffer fills completely.

• Inbound Drops (Rx):

  • The upstream physical Top-of-Rack (ToR) switch is sending data into the ESXi host faster than the physical adapter's Receive (Rx) ring buffers or the host CPU can process it.

Resolution

1. Verify  the nic stats and increase the ring buffer size to the maximum supported preset value as per Large number of "Receive packets dropped" & "Transmit packets dropped" on ESXi NIC's.
   
   Note: This needs to be performed for the network adapters across all the ESXi hosts in the vSAN Cluster. Increasing ring buffers can help mitigate dropped network packets (such as spikes in the Outbound Packet Drop Rate of pNIC vSwitch Port metric) by providing the ESXi hypervisor with a larger local queue to absorb network traffic microbursts before packets are discarded.

2. Verify the Drop Counters and Pause Frames 

   1. esxcli network nic stats get -n vmnicX

   2. esxcli network nic stats get -n vmnicX | grep -i pause

      If Receive pause frames are actively incrementing, the upstream physical switch is explicitly telling the ESXi host to stop sending data. This proves the bottleneck is on the physical network fabric, not the hypervisor.

3. If maximizing the ESXi ring buffers does not resolve the pNIC vSwitch Port Drop Rate, or if ESXi is actively receiving Rx Pause Frames, the virtual infrastructure is operating correctly but is waiting on the physical network.
   Engage your Network team or the hardware switch vendor and verify the below:

   1. Flow Control Mismatches:
      
      Audit the Top-of-Rack (ToR) or Spine switches. Ensure Priority Flow Control (PFC) or Link-Level Flow Control (Pause Frames) is configured symmetrically across the entire VLAN or fabric.
      Aggressive or mismatched flow control will instantly induce ESXi port drops.

   2. Switch Port Buffer Exhaustion:

      Ask the vendor to check the switch ASIC dynamic port buffer allocations.
      If the physical switch has shallow buffers, it will be unable to absorb storage or application microbursts, resulting in silent packet discards or excessive pause frame generation.

   3. Physical Hardware Defects:
      
      Request the network team to review the physical switch interfaces connected to the affected ESXi hosts for hardware-level faults (e.g., CRC errors, faulty transceivers, or failing twinax/fiber cables) that could be throttling the link capacity.