VMs Freeze and Host Management Deadlocks Due to iSCSI Path Loss and Port Group Misconfiguration During Link Flaps

Products

• VMware vSphere ESXi

Issue/Introduction

• Host management service (hostd) becomes unresponsive, causing the host to appear disconnected in vCenter Server.
• VMs crash or power off abruptly with page fault errors recorded in vobd.log:
  [vob.vm.kill.unexpected.vmx.fault.failure] The virtual machine using the configuration file /vmfs/volumes/.../VM.vmx could not fault in a page for the virtual machine user world. The virtual machine has been powered off.
  [esx.problem.vm.kill.unexpected.vmx.fault.failure.2] ... could not fault in a page. The VM is terminated as further progress is impossible.
• Host reboot is required to restore management responsiveness and virtual machine services.
  In vmkwarning.log and hostd.log, long I/O delay warnings and SCSI NMP probe warnings appear prior to the outage:
  WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:235: NMP device "naa.#####..." state in doubt; requested fast path state update...
  WARNING: ScsiDeviceIO: 13030: READ CAPACITY on device "naa.#####..." from Plugin "NMP" failed. Limit exceeded
  [Originator@6876 sub=IoTracker] In thread 2098812, fopen("/vmfs/volumes/.../VM.vmx") took over 113 sec.
• Verify iSCSI Port Group Uplink Configuration
  Review the Standard vSwitch configuration. Software iSCSI port binding requires 1:1 mapping between VMkernel adapters and physical uplinks:

Non-Compliant Configuration (Observed):
PortGroup Name    VLAN ID    Uplinks
iSCSI-1           #          vmnic#
iSCSI-2           #          vmnic#, vmnic##   <-- Incorrect: Multiple active uplinks

• Check vobd.log for Link Loss and iSCSI Session Timeout
  Look for network link drops followed by SCSI path dead state events:
  [vob.net.pg.uplink.transition.down] Uplink: vmnic2 is down. Affected portgroup: iSCSI-2. Failed criteria: 128
  [vob.iscsi.session.recovery.timeout] iScsi session recovery timeout for vmhba64:C0:T0
  [vob.scsi.scsipath.pathstate.deadver2] scsiPath vmhba64:C0:T0:L0 changed state from on
  [esx.problem.storage.redundancy.lost] Lost path redundancy to storage device naa.#####...

Environment

VMware vSphere ESXi 8.x

Cause

This issue is caused by a combination of a physical network link failure and an improper Software iSCSI Port Group configuration:

Physical Link Flapping (Hardware Layer): The primary physical uplink (vmnic#) experienced repeated physical link loss (Failed criteria: 128 in vobd.log, indicating a Physical Link State Loss / Carrier Off).

iSCSI Port Group Misconfiguration: The vSwitch software iSCSI port groups were configured with multiple active uplinks on the same port group (iSCSI-2 had both vmnic# and vmnic## active), violating VMware software iSCSI binding requirements.

Storage I/O Hang & Deadlock: When vmnic# went down, the misconfigured iSCSI pathing failed to fail over cleanly, leading to an all-paths-down (APD) / storage I/O deadlock on the VMFS datastore.

VM Crashes & Management Hang: Because active VM backing files (VMDK/VMX) became unreadable on the hung storage layer, VMs encountered unrecoverable memory page faults and crashed. Concurrently, hostd hung waiting on blocked storage file operations (fopen/realpath), making the host unresponsive.

Resolution

Action 1: Correct the Software iSCSI Port Group Uplink Teaming
Please refer to KB#317719

Action 2: Replace Faulty Physical Network Hardware (SFP+ Module / Cable)
Please refer to KB#426494

Action 3: Update Network Interface Driver (i40en)
Update the Intel i40en network driver to the latest recommended release to ensure driver-level link-handling stability.