ESXi hosts becomes unresponsive following FC link issue and repeated H:0x1 SCSI codes

Products

• VMware vSphere ESXi
• VMware vSphere ESX 7.x

Issue/Introduction

• A number of ESXi hosts in a cluster become unresponsive in vCenter. 
  
  
• Reboot is needed to restore manageability of the hosts. 
  
  
• Prior to the host becoming unrresponsive, vSphere Client reports:
  
  • hostd performance has degraded due to high system latency…
• While the hosts are unresponsive VMs continue to run
  
  

Environment

VMware vSphere ESXi 7.0.3

Cause

A change/temporary error is reported on the FC Link on the ESXI hosts. This triggers repeated no connection (H:0x1) warnings on paths to a specific target or targets. 

Sufficient connectivity to the target remains that the device does not enter a Permantent Device Loss state.

Repeated connecitivity issues over a period of time may eventually cause some ESXi hosts to become unresponisve.


/var/log/vmkernel.log reports FC link error:

WARNING: hfcldd1: HFC_EVNT2 FC Adapter Link Changed (ErrNo:0x18)
WARNING: hfcldd1: HFC_EVNT3 FC Adapter Driver Warning Event (ErrNo:0x18)
WARNING: hfcldd1: HFC_ERR6 Temporary FC Link error (ErrNo:0x83)
WARNING: hfcldd1: HFC_ERR6 Temporary FC Link error (ErrNo:0x0c)
WARNING: hfcldd1: HFC_EVNT3 FC Adapter Driver Warning Event (ErrNo:0x0c)

(Logging will vary depending on the HBA driver).

/var/log/vmkernel.log reports repeated H:0x1 for specific paths:
NMP: nmp_ThrottleLogForDevice:3867: Cmd 0x0 (0x45da196f4648, 0) to dev "naa..################################" on path "vmhba2:C0:T12:L##" Failed:
NMP: nmp_ThrottleLogForDevice:3875: H:0x1 D:0x0 P:0x0 . Act:NONE. cmdId.initiator=0x4539ca61bc18 CmdSN 0x0
NMP: nmp_ThrottleLogForDevice:3867: Cmd 0x0 (0x45ba2d41a508, 0) to dev "naa..################################" on path "vmhba2:C0:T12:L##" Failed:
NMP: nmp_ThrottleLogForDevice:3875: H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0. Act:NONE. cmdId.initiator=0x4539c569bb98 CmdSN 0x0

All impacted paths on a host to different devices are to the same target (in this case T12)

Resolution

Investigate on the fabric and storage, with assistance of fabric and storage vendors as required. 

Additional Information

For a broadly similar issue, see: ESXi host becomes unresponsive following repeated PDL errors on paths