Phantom write or read latency on a vSAN disk group

Products

• VMware vSAN

Issue/Introduction

Monitoring tools may report high average write or read latency on a vSAN disk group despite no actual performance degradation or virtual machine impact. This behavior, often referred to as "phantom latency," typically occurs during periods of extremely low I/O activity.

Symptoms:

• High average read or write latency observed at the vSAN disk group level.
• Latency spike observed on a cache disk leading to the reported disk group latency.
• In the vSphere UI, at the VM level, navigating to Monitor > vSAN > Performance, you may observe high read/write latency with minimal or no I/O activity, as shown in the performance graph below.

 

• Latency can also be observed at the disk layer (write buffer write latency) when cluster and vsan datastore is empty:

• No reported impact to virtual machine performance or guest operating system stability.

Environment

VMware vSAN 8.x

VMware vSAN 9.x

Cause

Measuring I/O latency requires I/O activity to measure against.  If it does not exist, then measurements can occasionally be misleading. If there were not enough read I/Os across that sampling period to measure the latency against, thus generating a misleading number.  This low number of I/O, sometimes referred to as "trickle I/O" can occasionally generate false latency spikes on any type of shared storage. This phenomenon is a reporting artifact known as phantom latency and does not indicate a hardware failure or operational bottleneck.

Resolution

To verify and confirm the latency is a reporting artifact, perform the following steps:

1. Navigate to Monitor > vSAN > Performance in the vSphere UI.
2. Review the Disk Group and Virtual Machine consumption graphs.
3. Confirm that IOPS and throughput were zero or near-zero during the reported latency spike.
4. Examine the host logs using a CLI or by reviewing support bundles.
5. Verify that /var/log/vmkernel.log and /var/log/vobd.log contain no storage resets, SCSI timeouts, or hardware sense codes during the timestamp of the alert.
6. If no VM impact is observed and logs are free of functional errors, the alert can be safely ignored.

 

Additional Information

If you require further assistance or need to speak with a Support Engineer, see Contact Broadcom support. Scroll to the bottom of the page and click on your respective region.