ESXI Host showing disconnected from Vcenter and VM not responding

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• ESXI host  is disconnected from the vCenter.
• There is no heartbeat between the vCenter and Host
• Host is marked as disconnected from the vCenter.
• On the ESXi DCUI console, you may observe the following error (the value may vary).
  IssueCommand:ERROR Tag 1 SActive already set: SACI:3E CI:3E activeTags:0 reissue_flag:0
•  On the vpxd logs, the error below was seen.
  error vpxd cannot contact the specified host (xxxxxx)
• On ESXi Cannot view the vmfs/volumes partition
• In the live vmkernel logging accessed via the console ( e.g., iDRAC, Alt+F12), the following errors may be observed:

<YYYY-MM-DD>T<HH:MM:SS> cpu39:#######)HPP: HppAttemptFailoverRequest:1391: Re-issuing first command for HPP device "t10.ATA_____ThinkSystem_M.2_VD______________________########################" (NO_CONNECT_ON_APD = CLEAR)

Or 

<YYYY-MM-DD>T<HH:MM:SS> cpu45:#######)HPP: HppAttemptFailoverRequest:1391: Re-issuing first command for HPP device "t10.ATA_____DELLBOSS_VD_____________________________########################" (NO_CONNECT_ON_APD = CLEAR)

Environment

vSphere ESXi 7.x

Cause

It is a known issue on Marvell 88SE9230 AHCI chipset based controllers (including DELL BOSS S1 adapter and Lenovo ThinkSystem_M.2 adapter).

Resolution

• Engage your hardware vendor.
• Along with hardware vendor support, you can embark on the below recommendations.
  1) Replace two SSDs with new same model SSDs. 
  2) Don't run VM or heavy IO on the ThinkSystem_M.2 device with RAID mode.
  3) If possible, configure the controller to JBOD mode. JBOD mode is fine enough as a boot device without running VM on it.