Failed to perform reboot from Powerflex Appliance "PowerFlex mapVol IO_Do_CK.. Retrying IO Type Read. Failed"

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• ESXi hosts installed on PowerFlex appliance are unable to reboot when the reboot command is initiated from vCenter through SSH or directly from the PowerFlex management interface.
• Messages in ESX - var/run/log/vmkwarning.log shows sockets closing - "Error Connection reset by peer to shutdown socket 0x4314c278de00"

[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu4:2100417)WARNING: PowerFlex netSock_RcvIntrn:1925 :Error: Failed Connection reset by peer to receive 128 data PTR 0x45b9d4749420 socket 0x45b9d4749b30
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu4:2098394)WARNING: PowerFlex netSock_Disconnect_CK:1065 :Warning: Error Connection reset by peer to shutdown socket 0x4314c278de00
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu22:2100465)WARNING: PowerFlex netSock_RcvIntrn:1925 :Error: Failed Connection reset by peer to receive 128 data PTR 0x45b9d473ef20 socket 0x45b9d473f630
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu4:2098394)WARNING: PowerFlex netSock_Disconnect_CK:1065 :Warning: Error Connection reset by peer to shutdown socket 0x4314c278e3a0
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu76:2100585)WARNING: PowerFlex netSock_RcvIntrn:1925 :Error: Failed Connection reset by peer to receive 128 data PTR 0x45b9d47420a0 socket 0x45b9d47427b0
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu4:2098394)WARNING: PowerFlex netSock_Disconnect_CK:1065 :Warning: Error Connection reset by peer to shutdown socket 0x4314c278ed60
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu65:2100586)WARNING: PowerFlex netSock_RcvIntrn:1925 :Error: Failed Connection reset by peer to receive 128 data PTR 0x45b9d4735aa0 socket 0x45b9d47361b0

• There are also messages in hostd.log that indicate hardware events in var/run/log/hostd.log

[YYYY-MM-DDTHH:MM:SS] In(166) Hostd[2101175] --> }
[YYYY-MM-DDTHH:MM:SS] In(166) Hostd[2101190] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 66 : Host had been powered off. The poweroff was not the result of a kernel error, deliberate reboot, or shut down. This could indicate a hardware issue. Hardware may reboot abruptly due to power outages, faulty components, and heating issues. To investigate further, engage the hardware vendor.
[YYYY-MM-DDTHH:MM:SS] In(166) Hostd[2101208] [Originator@6876 sub=Libs opID=3fb79223 sid=52cb0c1b] IOFilterInfoImpl: Inbox-IOFilter Id: VMW_spm_1.0.0, localId: spm
[YYYY-MM-DDTHH:MM:SS] In(166) Hostd[2101208] [Originator@6876 sub=Libs opID=3fb79223 sid=52cb0c1b] IOFilterInfoImpl: Inbox-IOFilter Id: VMW_vmwarevmcrypt_1.0.0, localId: vmwarevmcrypt

DCUI Screenshot while reboot from PowerFlex:

Environment

VMware ESXi Version: 8.0.x.

Cause

BIOS Issue. During the reboot of the ESXi from the PowerFlex appliance, an interruption occurred in I/O operations, which adversely affected access to the storage volumes.

 

 

Resolution

Kindly engage with the hardware vendor to address the issue promptly.

Please reference hardware article below:

https://www.dell.com/support/kbdoc/en-us/000067874/vxflex-14g-ready-node-spontaneous-sdc-disconnects