Cluster performance degrades due to high disk IO latency on single drive

Products

• VMware vSAN
• VMware Cloud Foundation

Issue/Introduction

• VM running slow or not responding
• A vSAN OSA cluster environment encounters sudden and severe performance degradation.
• "esxcli vsan health cluster list " command will explicitly flag this as a "Physical disk issue" with a Red status.
• vmkernel logs on the ESXi host showing these messages coming from single drive:
  

  [YYYY-MM-DDTHH:MM:SS] cpu51:2098041)WARNING: ScsiDeviceIO: 513: Device naa.########## performance has deteriorated. I/O latency increased from average value of 38762 microseconds to 776315 microseconds.

• Unexpected resync activity visible in Cluster > Monitor > vSAN > Resyncing Objects despite no manual maintenance or configuration changes. It occurs due to the Dying Disk Handling (DDH) taking the disk/disk group out of service. 

Environment

VMware vSAN 7.x (OSA)
VMware vSAN 8.x (OSA)
VCF:4.5.2

Cause

A degraded physical disk drive claiming capacity or caching inside a vSAN disk group fails to process IO blocks efficiently, creating an internal command queue backlog that spikes storage latency.

Resolution

1. Establish an SSH session to the affected ESXi host using root credentials.

2. Identify the device (naa.#####) identifier for the degraded disk. This can be obtained from /var/log/vmkernel.log showing the degraded messages

3. Validate that data accessibility can be maintained during removal by running the evacuation precheck command:

esxcli vsan debug evacuation precheck

4. Once the precheck confirms that the operation is safe and data loss will not occur, execute the following command to completely remove the disk from the configuration:

esxcli vsan storage remove -d naa.######

5. Monitor the host storage paths to ensure the cluster stabilizes and latency drops back to baseline thresholds.