ESXi host not responding: iSCSI Task mgmt stuck and heartbeat timeouts (race condition in iscsi_vmk)

Products

• VMware vSphere ESXi

Issue/Introduction

• This KB is for awareness on this issue impacting ESXi hosts on multiple versions connected to iSCSI storage via the vmk_iscsi software driver to storage devices being presented from multiple iSCSI storage vendors.

• ESXi Hosts enter a not responding state due to underlying storage issues.
• 'Waiting for timed out HB' messages related to a VMFS datastore connected to the hosts through software based vmk_iscsi adapter may be present in the '/var/log/vmkernel.log' on the ESXi host. These timeouts continue for a long time without any indication of scsi sense code that indicate an ongoing issue. 
• If the ESXi host is running 6.7 or higher, indications of a stuck task mgmt command may be present. For example:
  
  

  Task mgmt request issued to device naa.################################ is stuck

  
   

Environment

• VMware vSphere ESXi 6.x
• VMware vSphere ESXi 7.x
• VMware vSphere ESXi 8.x
• Software iSCSI

Cause

• This is due to a race condition inside iscsi_vmk driver, commands and the corresponding task management process can get stuck at the driver level. 

Resolution

• Fixed in release ESXi 6.5 P05, 6.7 P07, 7.0 U3f and higher. See Download Broadcom products and software for steps to download these releases.
  
  
• If this issue is encountered on ESXi 8.x, open a support case with Broadcom Support for further investigation.

Additional Information

• Impact/Risks:
• • ESXi hosts enter a 'not responding' state after Switch upgrades or Storage node reboots.
  • Once a host enters a not responding state you will lose manageability over the Host and the VMs which reside within.