High latency on all vSAN disks

Products

• VMware vSAN

Issue/Introduction

Symptoms : 

• VM's running on affected disk group reported performance issue. 
• vSphere UI indicates high latency associated with Disk Group cache device. 
• The ESXi host displays elevated latency metrics.

• The following, “PowerOn Reset” messages can be found in the hostd.logs :

2025-04-15T15:03:20.177Z info hostd[2105591] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 1250 : Frequent PowerOn Reset Unit Attentions are occurring on device naa.xxxxxxxxxxxxxxxx. This might indicate a storage problem. Affected datastores: Unknown

• The vmkernel logs recorded SCSI Write (0x2a) command failures, which are indicative of low-level disk I/O errors:

2025-07-10T21:24:15.027Z vmkernel: Cmd 0x2a to dev "naa.XXXXXXXXX" failed — Valid sense data: 0x5 0x25 0x0

• The vobd logs recorded Multiple disks were reported as permanently inaccessible

2025-07-03T18:12:07.028Z vobd[2097956]: [vob.scsi.device.state.permanentloss] Device :naa.XXXXXXXX has been removed or is permanently inaccessible.2025-07-10T11:53:40.922Z vobd[2097956]: [vob.scsi.device.state.permanentloss] Device :naa.XXXXXXXX has been removed or is permanently inaccessible.2025-07-10T12:23:11.395Z vobd[2097956]: [vob.scsi.device.state.permanentloss] Device :naa.55cd2e4150de4b17 has been removed or is permanently inaccessible

Environment

VMware vSAN 7.x

VMware vSAN 8.x

Cause

A hardware issue with the disk group cache device led to increased latency and permanent device loss. The failures, confirmed by above SCSI command errors and device removal messages in logs, Above error indicate this issue is outside the ESXi host, indicating underlying physical or firmware-related disk instability.

Resolution

Contact hardware vendor for further assistance.