ESXi hosts periodically lose access to volumes presented to the hosts by DellEMC PowerFlex

Products

• VMware vSphere ESXi
• VMware vSphere ESX 5.x
• VMware vSphere ESX 6.x
• VMware vSphere ESX 7.x
• VMware vSphere ESX 8.x
• VMware vSphere ESXi 5.0
• VMware vSphere ESXi 5.5
• VMware vSphere ESXi 8.0

Issue/Introduction

An Administrator will observe "Lost access to volume" events in vCenter, VMs may also crash or go unresponsive

Environment

ESXi (All versions)

DellEMC PowerFlex

Cause

On the ESXi host, in /var/log/vmkernel.log will show entries from the DellEMC PowerFlex driver stating a "IO_HARD_ERROR", which translates to a hardware error:

2025-05-12T07:52:49.524Z cpu69:2098268)WARNING: PowerFlex mapVolIO_Do_CK:1510 :Mit: 0x45d9b80e2740. Vol ID 0x########00000003. Last fault Status IO_HARD_ERROR(20).Last error Status TIMEOUT(68) Retry count (5) chan (307)
2025-05-12T07:52:56.786Z cpu43:2098268)WARNING: PowerFlex mapVolIO_Do_CK:1510 :Mit: 0x45b9f049b3c0. Vol ID 0x########00000004. Last fault Status IO_HARD_ERROR(20).Last error Status TIMEOUT(68) Retry count (1) chan (307)
2025-05-12T07:52:57.090Z cpu43:2098268)WARNING: PowerFlex mapVolIO_Do_CK:1510 :Mit: 0x45b9ecaed340. Vol ID 0x########00000008. Last fault Status IO_HARD_ERROR(20).Last error Status TIMEOUT(68) Retry count (1) chan (10)

Alongside these DellEMC PowerFlex specific events from their driver, there will also be entries from ScsiDeviceIO stating CHECK CONDITION, HARDWARE ERROR:

2025-05-12T07:52:45.823Z cpu41:2098472)ScsiDeviceIO: 4176: Cmd(0x45d9a58c7788) 0x2a, CmdSN 0xffffb706217de5e0 from world 2116019 to dev "eui.########################00000003" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x0 0x0
2025-05-12T07:52:49.524Z cpu40:2098472)ScsiDeviceIO: 4176: Cmd(0x45d9ce0c2d08) 0x2a, CmdSN 0xffffb7061e294cb0 from world 2116019 to dev "eui.########################00000003" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x0 0x0
2025-05-12T07:52:49.524Z cpu40:2098472)ScsiDeviceIO: 4176: Cmd(0x45d998685288) 0x2a, CmdSN 0xffffb70615627690 from world 2116019 to dev "eui.########################00000003" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x0 0x0

Resolution

Contact DellEMC PowerFlex support for assistance in root causing these events. In this particular instance, there was a failed physical hard drive in the DellEMC PowerFlex solution.