High Latency on a vSAN disk group

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• High average read latency observed on a vSAN disk group.
• Latency spike observed on a cache disk leading to the reported disk group latency.
• In the vSphere UI, at the VM level, navigating to Monitor > vSAN > Performance, you may observe high read/write latency with minimal or no I/O activity, as shown in the performance graph below.

 

• Latency can be observed also on disk layer (write buffer write latency) when cluster and vsan datastore is empty:

Environment

VMware vSAN 7.x

VMware vSAN 8.x

Cause

The latency spike occurred without active I/O on the device, indicating phantom latency rather than an actual functional or operational issue.

Resolution

When there is little to no I/O activity on the disk it can occasionally generate false latency spikes on any type of shared storage. This phenomenon of phantom latency may be seen at the cluster level, or down to even the individual VMDK.

Phantom-latency may occur due to the way performance data is processed and graphed.

Such instances should be verified against supporting host logs by raising a ticket with Broadcom Support to confirm that they are not related to any functional or operational issue on the device.

Additional Information

Troubleshooting vSAN performance