High Disk Latency on vSAN cluster may impact VM performance.
search cancel

High Disk Latency on vSAN cluster may impact VM performance.

book

Article ID: 393495

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

Symptoms:

  • Virtual Machines running on the vSAN cluster may be in a hung state or not responding when launching VM console.

  • vSAN resync operations may appear frozen or stuck at a specific percentage.

  • vSAN disk groups intermittently show "0/0" capacity disks or "0" disk count in vCenter Disk Management.

  • Hardware console of the host may show all Physical Disks (PD) and Virtual Disks (VD) as "Online."

  • esxcli vsan debug resync list shows resync is frozen or objects are stuck syncing to a specific destination host.

  • When checking for the vSAN cluster performance, cluster level latency may report READ and WRITE latency.

  • VMkernel logs may report deteriorated performance for specific devices: WARNING: ScsiDeviceIO: 1779: Device naa.#### performance has deteriorated. I/O latency increased

  • High latency is reported for READ and WRITE operations at the cluster level.

  • vSAN Health check reports Congestion alarm with status Yellow or Red.
  • hostd logs record congestion threshold events: 

    yyyy-mm-ddT10:10:20.884Z info hostd[2104070] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 426919 : LSOM SSDCong in ######-###-###-###-##### Congestion State: Exceeded. Congestion Threshold: 200 Current Congestion: 204.

Environment

  • VMware vSAN 7.x

  • VMware vSAN 8.x

  • VMware vSAN 9.x

Cause

  • The issue is seen when there are one or more disks used for vSAN experiencing high latency from the physical layer.

  • Upon checking the vmkernel logs on ESXi hosts under /var/run/log/vmkernel.log, latency will report high.
  • Example events as below indicate the physical disk latency is around 3.9 seconds.

2026-03-13T06:25:40.216Z Wa(180) vmkwarning: cpu13:2097946)WARNING: ScsiDeviceIO: 1779: Device naa.#### performance has deteriorated. I/O latency increased from average value of 21153 microseconds to 3965595 microseconds.
2026-03-13T06:25:41.119Z Wa(180) vmkwarning: cpu13:2097946)WARNING: ScsiDeviceIO: 1779: Device naa.#### performance has deteriorated. I/O latency increased from average value of 21153 microseconds to 3710764 microseconds.
2026-03-13T06:25:42.159Z Wa(180) vmkwarning: cpu13:2097943)WARNING: ScsiDeviceIO: 1779: Device naa.#### performance has deteriorated. I/O latency increased from average value of 21153 microseconds to 3590664 microseconds.

  • When reviewing performance statistics under vSAN Cluster > Support > Performance > Diagnostics, latency may be observed for individual disks under the LSOM category.

Resolution

  1. To address to the issue immediately, after identifying the problematic host, place the host into maintenance mode using ensure accessibility or full data migration and monitor the performance on the vSAN cluster.

  2. If the performance improves, work with hardware vendor to get the hardware issues resolved.

  3. The following article can be used for instructions on Requirements when replacing disks in a vSAN cluster

  4. To identify the disks with the highest latency, the following command can be run on the vmkernel.log on an ESX host Putty session:
    grep deteriorated /var/log/vmkernel.log | awk '{print $22 " " $8}' | uniq | sort -n -r | less

    Sample output, left column is in microseconds and the right column is the device:
    1772015 naa.5000039c########
    1765728 naa.5000039c########
    1708035 naa.5000039c########
    1699681 naa.5000039c########
    1654993 naa.5000039c########
    1652807 naa.5000039c########

Note: Check the HBA driver and firmware combo on the Broadcom vSAN I/O Controller HCL as well as the drive firmware version.

Additional Information

vSAN performance might degrade further when accompanied by Reactive Rebalancing - an automated activity temporarily increasing I/O demand in the cluster at high-capacity usage levels (at or above 80%). This situation can be avoided by making sure that total usage never crosses 80% (75% for optimal overall health). If cluster is found to undergo automatic rebalance tasks frequently, it is recommended to decrease the total space usage by evacuating some load, cleanup activities and/or expanding storage capacity in the cluster.