vSAN ESA Virtual Machine Read and Write Latency Spikes During UNMAP or File Overwrite Workloads

Products

• VMware vSAN
• VMware vSphere ESXi

Issue/Introduction

When running latency-sensitive, high-frequency write/rewrite or UNMAP space reclamation workloads (such as trading session persistence, database logging, or rapid temporary file overwrites) on a vSAN Express Storage Architecture (ESA) cluster virtual machine disks may experience transient latency spikes.

• Virtual machine disk read and write latencies spiking between 400 ms and over 2,000 ms (2+ seconds).

• Performance degradation correlating directly with high-frequency, small-block, latency-sensitive write/rewrite workloads running concurrently with active TRIM/UNMAP space reclamation.
  
  

Environment

• VMware vSAN Express Storage Architecture (ESA) 8 All Versions
• VMware vSAN Express Storage Architecture (ESA) 9.1

Cause

A background space-cleanup process in vSAN ESA stalls during file deletion operations, causing incoming compute write operations to queue and temporarily freeze while waiting for space reclamation processing.

Resolution

Broadcom is working on a fix that will be included in an upcoming release. Please contact Broadcom support for further investigation and assistance.

To stabilize virtual machine write latency and prevent guest I/O from being impacted by UNMAP throttling, reduce the UNMAP throttle timeout parameters to 1 millisecond across all hosts in the vSAN ESA cluster.

Run the following commands via SSH 

esxcfg-advcfg -s 1 /VSAN/zDOMUnmapTimeoutMsec

esxcfg-advcfg -s 1 /VSAN/zDOMUnmapMaxTimeoutMsec

 

 

Additional Information

Subscribe to this Article for updates.