Esxi host boot stuck at "activating: software-iscsi"

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms : 

• ESXi host takes long time to boot.
• Boot screen will stuck at "activating: software-iscsi" screen for long time. 
  

Environment

VMware vSphere ESXi 6.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

• For iSCSI sendtarget discovery on ESX, ESXi will try to discover sendtargets with port bound vmknics (example: vmk1 and vmk2) and also with routable path (example VMK1, VMK2 and VMK0 here). 
• Discovering iscsi send targets through the default interface failing, leading to longer boot time.

Cause validation:

• Run the below command to validate the routable path for discovering iscsi send targets.
  

  esxcfg-route -l -N defaultTcpipStack

  VMkernel Routes:
Network          Netmask          Gateway          Interface
##.#.##.#        ###.###.###.###  Local Subnet     vmk2
##.#.###.###     ###.###.###.###  Local Subnet     vmk1
##.#.##.#        ###.###.###.###  Local Subnet     vmk0
default          #.#.#.#          ##.#.##.#        vmk0

• var/run/log/syslog.log file confirm that discovering iSCSI send targets through the default interface is failing.
  YYYY-MM-DDTHH:MM.SSSZ Db(31) iscsid[2099143]: discovery_sendtargets::Running discovery on IFACE default(iscsi_vmk) target addr=##.#.#.123:3260 (drec.transport=iscsi_vmk)
YYYY-MM-DDTHH:MM.SSSZ Wa(28) iscsid[2099143]: connection failed for discovery (err = Interrupted system call)!
YYYY-MM-DDTHH:MM.SSSZ Er(27) iscsid[2099143]: connection to discovery address ##.#.#.123 failed
YYYY-MM-DDTHH:MM.SSSZ Db(31) iscsid[2099143]: discovery_sendtargets::Running discovery on IFACE default(iscsi_vmk) target addr=##.#.#.456:3260 (drec.transport=iscsi_vmk)
YYYY-MM-DDTHH:MM.SSSZ Wa(28) iscsid[2099143]: connection failed for discovery (err = Interrupted system call)!
YYYY-MM-DDTHH:MM.SSSZ Er(27) iscsid[2099143]: connection to discovery address ##.#.#.456 failed
YYYY-MM-DDTHH:MM.SSSZ Db(31) iscsid[2099143]: discovery_sendtargets::Running discovery on IFACE default(iscsi_vmk) target addr=##.#.#.78:3260 (drec.transport=iscsi_vmk)
YYYY-MM-DDTHH:MM.SSSZ Wa(28) iscsid[2099143]: connection failed for discovery (err = Interrupted system call)!
YYYY-MM-DDTHH:MM.SSSZ Er(27) iscsid[2099143]: connection to discovery address ##.#.#.78 failed

Resolution

• Engage internal network team to ensure vmk0 can reach the discovery target IP addresses.

                                     Or

• Log in to the host via SSH and configure a static route for each discovery IP address to bypass vmk0 using command mentioned below:
  
  esxcfg-route -a <TARGET-IP>/32 <Gateway-IP>
  

  This will prevent ESXi from attempting sendtarget discovery through VMK0.

Workaround : Reduce iSCSI login timeout from 30 seconds to 5 seconds.