High vSAN Disk group latency on host with CPU overutilization

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• Elevated disk group latency observed on a single host within the vSAN cluster.
• On the ESXi host, the Monitor > vSAN > Performance view shows a sudden spike in the Delayed I/O Average Latency counter of a Disk Group.

 

 

Environment

VMware vSAN 8.x

 

Cause

In a vSAN environment, CPU Ready should be ideally near 0% and below 1% for vSAN worlds. When vSAN modules such as LSOM,DOM and NVMe are forced to wait for CPU (Ready state) due to host level CPU overutilization it can cause latency to be observed on the disk group level

Resolution

To resolve the vSAN disk group latency caused by CPU scheduling delays for vSAN processes, Consider  right-sizing the over-provisioned workloads to alleviate the underlying scheduling contention for CPU to decrease vSAN CPU Ready values. If immediate right-sizing is not feasible, migrate compute-heavy VMs to other hosts in the cluster with more available physical cores to alleviate the immediate localized bottleneck.