ESXi hosts lose iSCSI connectivity to Storage during vMotion or peak I/O.
Article ID: 443476
Updated On:
Products
Issue/Introduction
- ESXi hosts intermittently lose connectivity to iSCSI datastores.
- The issue primarily occurs during high-load tasks such as vMotion, Storage vMotion, or scheduled backup windows.
- Hosts may become unresponsive in vCenter (Grayed out) or the Host Client.
Environment
VMware vSphere ESXi 8.x
Cause
The issue is typically caused by Network Resource Contention. When iSCSI VMkernel adapters (vmknics) share the same physical uplinks (vmnics) ,high-bandwidth traffic like vMotion, the vMotion operations can saturate the physical link.
This causes iSCSI heartbeats to time out, leading the ESXi initiator to mark the path as "Offline."
- Log Errors (var/log/vmkernel.log) on ESXi host
####-##-##T##:##:##.966Z Wa(180) vmkwarning: cpu35:4067312)WARNING: ScsiDeviceIO: 13055: READ CAPACITY on device "naa.##########" from Plugin "NMP" failed. I/O error
####-##-##T##:##:##..474Z In(182) vmkernel: cpu11:2098192)ScsiDeviceIO: 7917: Waited for completion for all issued commands for partition naa.########:1, for 120 secs. 1 completions still awaited. Giving up !
####-##-##T##:##:##..693Z In(182) vmkernel: cpu17:2098192)ScsiDeviceIO: 7917: Waited for completion for all issued commands for partition naa.#########:1, for 120 secs. 1 completions still awaited. Giving up
Resolution
To resolve this issue, ensure dedicated bandwidth and low latency for iSCSI traffic:
- Network Separation (Recommended):
- Physically or logically separate iSCSI traffic from vMotion and Management traffic.
- Move iSCSI Vmkernel ports to dedicated physical uplinks (vmnics) that do not carry vMotion traffic.
- Verify Storage QoS Policies:
- Verify that the QoS policy is not causing intentional I/O throttling.
- Driver and Firmware Audit:
- Ensure the latest supported async drivers are installed.
- MTU Consistency:
- Verify that Jumbo Frames (MTU 9000) are configured consistently across the entire path (VMkernel, vSwitch, Physical Switch, and Storage Array) or reverted to MTU 1500 globally.
Additional Information
Feedback
