VMs Experience Degraded Performance in vSAN Cluster Environment.

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• VMs running on vSAN datastore facing high write latency.
• You may not see any latency observed on the vSAN cluster under monitor>performance tab on vCenter.
• From /var/run/log/vmkernel.log may not show you performance deteriorated messages for vSAN disks.

Environment

VMware vSAN 8.x

Cause

This issue can occur if there are any read/write errors on the vSAN disks.

Verify the same from /var/run/log/vmkernel.log:

Ex: 

2025-11-03T19:47:46.208Z In(182) vmkernel: cpu8:2098209)ScsiDeviceIO: 4479: Cmd(0x45be9f6cb4c0) 0x2a, CmdSN 0x878c0a65 from world 0 to dev "naa.55cd#######3c55" failed H:0xc D:0x0 P:0x0

2025-11-03T19:47:46.208Z Wa(180) vmkwarning: cpu11:2098272)WARNING: HPP: HppThrottleLogForDevice:1133: Cmd 0x28 (0x45b9e232bf00, 0) to dev "naa.55cd#######3c55" on path "vmhba1:C0:T6:L0" Failed:

2025-11-03T19:47:46.458Z Wa(180) vmkwarning: cpu8:2098209)WARNING: HPP: HppThrottleLogForDevice:1133: Cmd 0x28 (0x45be9f7948c0, 0) to dev "naa.55cd#######3c55" on path "vmhba1:C0:T6:L0" Failed:

2025-11-03T19:47:46.708Z In(182) vmkernel: cpu0:2097902)ScsiDeviceIO: 4451: Cmd(0x45be9f7948c0) 0x28, cmdId.initiator=0x430a089ba680 CmdSN 0x878c0a7e from world 0 to dev "naa.55cd#######3c55" failed H:0x8 D:0x0 P:0x0

2025-11-03T19:47:46.708Z In(14) vobd[2098017]:  [APDCorrelator] 608581146890us: [vob.storage.apd.start] Device or filesystem with identifier [naa.55cd#######3c55] has entered the All Paths Down state

 

The referenced hard disk will likely report a significant Read Error Count in its SMART diagnostic status.

 

Verify the same from below command output: esxcli storage core device smart get -d naa.##############

 

Ex:

SMART Data for Disk : naa.55cd#######3c55
Parameter                       Value  Threshold Worst  Raw
-----------------------------------------------------------
Health Status                    OK      N/A     N/A     N/A
Media Wearout Indicator          100     0       100     34106456
Read Error Count                 130     39      130     4294967295
Power-on Hours                   100     0       100     22102
Power Cycle Count                100     0       100     42
Reallocated Sector Count         99      1       99      8
Drive Temperature                100     0       100     16
Write Sectors TOT Count          100     0       100     34106456
Read Sectors TOT Count           100     0       100     32287137
Pending Sector Reallocation Ct   100     0       100     0
Uncorrectable Sector Count       100     0       100     0

Resolution

1. Engage the hardware vendor (OEM) to perform comprehensive diagnostics on the affected physical disk(s) exhibiting I/O errors.
2. Based on the vendor's diagnostic results, proceed with the replacement of any faulty or failing storage media as per their official recommendation.
3. Request the hardware vendor to verify and confirm that all vSAN-contributing disks are utilizing the latest certified and recommended firmware versions consistent with the vSAN Hardware Compatibility List (HCL) and the disk model.