Warning: 'Errors occurred on the disk(s) of a vSAN host' on vSAN cluster and host.

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• vSAN cluster and host may show the critical warning as 'Errors occurred on the disk(s) of a vSAN host' for one of the hosts.

• vSAN skyline health report vSAN health alarm 'Physical disk health retrieval issues' warning.

• vSAN cluster may show performance issue and failed to load the vSAN tabs on vCenter UI.

Environment

VMware vSAN 7.x

VMware vSAN 8.x

VMware vSAN 9.x

Cause

• The above symptoms show up when there are underlying vSAN disks experiencing hardware issue with one or more physical disks.

• vmkernel.log would show READ/WRITE failures upon checking the vmkernel.log on ESXi host in the path /var/run/log.

vmkernel.log
YYYY-MM-DDTHH:MM:SS.SSZ cpu7:2098282)ScsiDeviceIO: 4052: Cmd(0x459e40892040) 0x28, cmdId.initiator=0x43053bca09c0 CmdSN 0x1 from world 2100443 to dev "naa.50000#######" failed H:0x5 D:0x0 P:0x0 Aborted at driver layer. Cmd count Active:4
YYYY-MM-DDTHH:MM:SS.SSZ cpu15:2098282)NMP: nmp_ThrottleLogForDevice:3856: Cmd 0x28 (0x459e29150a00, 0) to dev "naa.50000######" on path "vmhba0:C0:T7:L0" Failed:
YYYY-MM-DDTHH:MM:SS.SSZ cpu15:2098282)ScsiDeviceIO: 4062: Cmd(0x459e29150a00) 0x28, CmdSN 0xb7b40a from world 0 to dev "naa.50000######" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x1.
YYYY-MM-DDTHH:MM:SS.SSZ cpu0:2098282)ScsiDeviceIO: 4052: Cmd(0x459e7060f680) 0x28, cmdId.initiator=0x451a1151a798 CmdSN 0xb7b549 from world 0 to dev "naa.50000######" failed H:0x5 D:0x0 P:0x0 Aborted at driver layer. Cmd count Active:2
YYYY-MM-DDTHH:MM:SS.SSZ cpu31:2098282)ScsiDeviceIO: 4052: Cmd(0x459e409bc240) 0x28, cmdId.initiator=0x43053bced580 CmdSN 0x2 from world 2100443 to dev "naa.50000######" failed H:0x5 D:0x0 P:0x0 Aborted at driver layer. Cmd count Active:3

• Checking the hardware console for the server would show disks failure or predictive failure for the drives as below.

Predictive failure reported for Disk 6 in Backplane 1 of Integrated Storage Controller 1. Part Number

• Storage events from the server hardware console UI would show as below.

Resolution

1. Contact the hardware vendor to get the affected drive replaced.

2. After the hardware is replaced, the instructions in the below articles can be used to replace the appropriate drive on vSAN cluster.

Requirements when replacing disks in a vSAN cluster

Replace a Flash Caching Device on a Host in vSAN Cluster

Replace a Capacity Device in vSAN Cluster