iSCSI All Paths Down (APD) and H:0x1 (NO_CONNECT) Errors: Duplicate IP Address Conflict
Article ID: 440474
Updated On:
Products
Issue/Introduction
When utilizing software iSCSI for storage connectivity, a single ESXi host in a cluster may experience sudden, isolated storage disconnects while other hosts remain unaffected. You may observe the following symptoms:
vCenter Server Alerts: Datastores go offline and trigger "Lost connectivity" alarms.
Example:
Cannot connect to storage on esxi-host-01.local triggered by event 'Lost connectivity to storage device naa.6#########################. Path vmhba64:C0:T0:L1 is down. Affected datastores: iSCSI_Datastore_01.'
Host Log Errors (
/var/log/vmkernel.log): The Native Multipathing (NMP) plugin reports constant connection drops and host busy states.Example:
NMP: nmp_ThrottleLogForDevice:3898: H:0x1 D:0x0 P:0x0 . Act:NONE.(NO_CONNECT)Example:
NMP: nmp_ThrottleLogForDevice:3898: H:0x2 D:0x0 P:0x0 . Act:EVAL.(HOST_BUSY)
Manual Rescan Failures: Forcing a storage adapter rescan triggers an immediate All Paths Down (APD) event due to the host failing to retrieve SCSI INQUIRY data.
Example:
StorageApdHandler: 1051: Freeing APD handle...Example:
VMW_SATP_ALUA: satp_alua_getTargetPortInfo:190: Could not get page 83 INQUIRY data for path "vmhba64:C0:T0:L1" - No connection
Environment
VMware vSphere ESXi 8.x
VMware vSphere ESXi 9.x
Cause
This issue occurs when there is an IP Conflict (Duplicate IP Address) on the iSCSI storage network.
This typically occurs when another device (e.g., a VM with guest-attached iSCSI or another physical host) is assigned the same static IP address as the ESXi host's dedicated iSCSI VMkernel port. This causes ARP thrashing on the physical switch, leading to misrouted packets and dropped TCP sessions
Resolution
To resolve this issue, you must identify and reconfigure the conflicting device to eliminate the ARP thrashing on the storage fabric.
Identify the Conflict: Check the physical switch's ARP table for the ESXi host's VMkernel IP address to see if the MAC address is rapidly flapping between two different devices.
Alternatively, temporarily disable the affected VMkernel port on the ESXi host and run a continuous ping to that IP address from another machine on the storage subnet. If the ping succeeds while the ESXi host port is down, another device is actively using that IP.
Reconfigure the Conflicting Device: Locate the Virtual Machine or physical device utilizing the duplicate IP and re-assign it a unique, available IP address within the storage subnet.
Restore Host Connectivity: * Ensure the ESXi host's VMkernel port is enabled.
Perform a manual storage rescan on the ESXi host via the vSphere Client or CLI (
esxcli storage core adapter rescan --all).Verify in the
/var/log/vmkernel.logthat paths are restored and no furtherH:0x1errors are generated.
Additional Information
For additional information on troubleshooting iSCSI connectivity and understanding SCSI NMP status codes, please refer to the following official Broadcom documentation:
Feedback
