VM Performance Degradation and Hard Resets due to vSAN Storage Device Internal Hangs

Products

VMware vSAN

Issue/Introduction

Symptoms:

Virtual Machines (VMs) running on vSAN datastores may experience severe performance degradation or unexpected hard resets. This is primarily seen in latency-sensitive applications where the Guest OS triggers a reset if storage I/O is delayed beyond internal thresholds.

Example : Latency-sensitive application workloads VMs (e.g., Oracle DB, Microsoft SQL Server)
Impacted VMs encounter high write latency or sudden reboots. In many cases, vCenter performance charts may not show the latency if the hang is intermittent and occurring at the hardware layer.

Example : vmware.log : 2026-05-14T08:22:59.646Z In(05) vcpu-2 - Chipset: The guest has requested that the virtual machine be hard reset.
You may not see any latency observed on the vSAN cluster under monitor>performance tab on vCenter.
From /var/run/log/vmkernel.log may not show you performance deteriorated messages for vSAN disks.
High Read and Write latency observed on a specific ESXi host affecting multiple virtual machines simultaneously.
Storage devices may intermittently stop responding to I/O commands (e.g., opcodes 0x28 or 0x88), followed by an automated hardware-level reset.
vSAN capacity disks report Power-on Reset events in vobd.log for the disk which has vSAN Data component for impacted VM.

vobd.log:

2026-05-14T08:22:34.595Z In(14) vobd[2097956]: [scsiCorrelator] 471056881431us: [vob.scsi.scsipath.por] Power-on Reset occurred on naa.xxxxxxxxxxxxxxxx
Physical disk health and SMART data may appear "OK" or healthy, despite the occurrence of internal device hangs.
vSAN traces indicates there is 'Wait for RDT' events which confirms there is transient network issue. vsantraces: 2026-05-14T08:20.439638 [3612038] [cpu22] [c826378b OWNER readWithBlkAttr5 VMDISK] DOMTraceOpTookTooLong:10304: {'op': 0x45bad8d63c80, 'objUuid': 'xxxxxxx-xxxxxx-xxxx-xxxx-xxxxxxxxxxxx', 'offset-39': 138322706432, 'length-25': 65536, 'totalTimeMS': 10014, 'timeInThisPhaseMS': 10014, 'opPhase': 'Wait for RDT'}

Environment

VMware vSAN 8.x

Cause

Physical storage devices intermittently stop responding to I/O commands, leading to a Storage Device Internal Hang. This triggers an automated hardware-level reset (Power-on Reset). The resulting "dead time" during the reset causes severe I/O delays. Latency-sensitive Guest OS configurations interpret this as a storage loss and initiate an automated reboot to recover.
This issue can occur if there are any read/write errors on the vSAN disks.

Verify the same from /var/run/log/vmkernel.log:

Ex:

2025-11-03T19:47:46.208Z In(182) vmkernel: cpu8:2098209)ScsiDeviceIO: 4479: Cmd(0x45be9f6cb4c0) 0x2a, CmdSN 0x878c0a65 from world 0 to dev "naa.55cd#######3c55" failed H:0xc D:0x0 P:0x0

2025-11-03T19:47:46.208Z Wa(180) vmkwarning: cpu11:2098272)WARNING: HPP: HppThrottleLogForDevice:1133: Cmd 0x28 (0x45b9e232bf00, 0) to dev "naa.55cd#######3c55" on path "vmhba1:C0:T6:L0" Failed:

2025-11-03T19:47:46.458Z Wa(180) vmkwarning: cpu8:2098209)WARNING: HPP: HppThrottleLogForDevice:1133: Cmd 0x28 (0x45be9f7948c0, 0) to dev "naa.55cd#######3c55" on path "vmhba1:C0:T6:L0" Failed:

2025-11-03T19:47:46.708Z In(182) vmkernel: cpu0:2097902)ScsiDeviceIO: 4451: Cmd(0x45be9f7948c0) 0x28, cmdId.initiator=0x430a089ba680 CmdSN 0x878c0a7e from world 0 to dev "naa.55cd#######3c55" failed H:0x8 D:0x0 P:0x0

2025-11-03T19:47:46.708Z In(14) vobd[2098017]:  [APDCorrelator] 608581146890us: [vob.storage.apd.start] Device or filesystem with identifier [naa.55cd#######3c55] has entered the All Paths Down state

The referenced hard disk will likely report a significant Read Error Count in its SMART diagnostic status.

Verify the same from below command output: esxcli storage core device smart get -d naa.##############

Ex:

SMART Data for Disk : naa.55cd#######3c55
Parameter Value Threshold Worst Raw
-----------------------------------------------------------
Health Status OK N/A N/A N/A
Media Wearout Indicator 100 0 100 34106456
Read Error Count 130 39 130 4294967295
Power-on Hours 100 0 100 22102
Power Cycle Count 100 0 100 42
Reallocated Sector Count 99 1 99 8
Drive Temperature 100 0 100 16
Write Sectors TOT Count 100 0 100 34106456
Read Sectors TOT Count 100 0 100 32287137
Pending Sector Reallocation Ct 100 0 100 0
Uncorrectable Sector Count 100 0 100 0

Resolution

Engage the hardware vendor (OEM) to perform comprehensive diagnostics on the affected physical disk(s) exhibiting I/O errors.
Based on the vendor's diagnostic results, proceed with the replacement of any faulty or failing storage media as per their official recommendation.
Request the hardware vendor to verify and confirm that all vSAN-contributing disks are utilizing the latest certified and recommended firmware versions consistent with the vSAN Hardware Compatibility List (HCL) and the disk model.
Investigate underlying network stability to ensure transient connectivity issues are not contributing to storage timeouts.