High storage latency due to SCSI WRITE SAME offload spikes on high-utilization storage arrays

Products

• VMware vSphere ESXi

Issue/Introduction

• A sudden, sharp increase in storage array I/O latency impacts virtual machines and host workloads across one or more vSphere clusters.

• Storage array telemetry indicates controller CPU utilization spiking to 100% saturation.

• SAN fabric, host HBAs, physical switch ports, and array hardware remain fully healthy with zero recorded hardware failures, path drops, or failovers.

• Storage performance metrics show a significant burst of hardware-accelerated SCSI WRITE SAME commands (Opcode 0x93) coinciding with the latency event.
  
  

• Standard ESXi host logs (vmkernel.log) show no path errors or storage dropouts, as VAAI / vVol storage offload commands are managed directly between the ESXi storage stack, Protocol Endpoint (PE), and the VASA Provider/Array Controllers

Environment

• VSphere Esxi 8.x
• VVOLs 

 

Cause

If the storage array is already operating at high baseline utilization (e.g., ~90–95% CPU load), the sudden flood of CPU-intensive block-zeroing requests exhausts all remaining controller headroom. Because controller CPUs become 100% saturated processing the offload commands, incoming routine I/O requests from adjacent host workloads are forced to queue, resulting in array-wide latency spikes and degraded application throughput.

Resolution

1. Work with the storage administration team to rebalance workloads across array controllers or expand array controller capacity.

2. Ensure storage array baseline CPU utilization is maintained at a healthy threshold (ideally keeping average baseline CPU load below 80%) to accommodate unexpected storage allocation bursts.

3. For workloads known to perform rapid, massive storage allocations in a short window, consider pre-allocating disk space (Eager Zeroed Thick) during scheduled maintenance windows. This transfers the block-zeroing overhead away from peak operational hours