ESXi hang due to local RAID/HDD error

Products

• VMware vSphere ESXi

Issue/Introduction

ESXi hang with following symptoms:

• Use commands like 'tail -f vmkernel.log', 'grep xxx vmkernel.log' to check logs will have no any output and hang.
• vmkernel has repeatedly following traceback:
  vmk_LogBacktraceMessage
  ahciRequestIo
  scsiExecReadWriteCommand
  ataIssueCommand
  scsiQueueCommand
• Local datastore heartbeat timeout:
  vobd.log:
  In(14) vobd[xxxxxxx]:  [vmfsCorrelator] 44543460374us: [vob.vmfs.heartbeat.timedout] <datastore_uuid> <datastore_name>
  vmkernel.log:
  Wa(180) vmkwarning: cpuxx:xxxxx)WARNING: HPP: HppScsiThrottleLogForDevice:585: Cmd 0x2a (0x45bdc1a9aa00, 2097224) to dev "t10.ATA_____XXXXXXXX_____________________________xxxxxxxxxxxxx" on path "vmhba3:C0:T0:L0" Failed: Error status H:0x5 D:0x22 P:0x0 Invalid sense data: 0x0 0x0 0x0. hppAction = 3
• ESXi lost controller connection with NSX manager.
• ESXi fail to enter maintenance mode due to VM vmotion fail.

Environment

vSphere ESXi

Cause

The local RAID/HDD error resulted in IO failure and brought ESXi in hang status, eventually caused VM vmotion failure, and NSX UI/Controller down.

Resolution

Contact the server vendor to troubleshoot hardware issues related to the RAID Controller and local disk.

Reboot ESXi may temporarily recover ESXi's operation.