iSCSI datastore path was down with error "Lost access to volume due to connectivity issues"

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• iSCSI LUN connectivity issue on ESX/ESXi with error:

  "Lost access to volume XXX due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly"
  

  

Environment

VMware ESXi 7.x

VMware ESXi 8.x

Cause

• In the vmkernel logs, it can be seen that the iSCSI connections are frequently being marked as "OFFLINE" and "ONLINE".

  YYYY-MM-DDT##:##:##.#### cpu67:2099071)WARNING: iscsi_vmk: iscsivmk_StopConnection:738: vmhba64:CH:2 T:0 CN:0: iSCSI connection is being marked "OFFLINE"
  YYYY-MM-DDT##:##:##.#### cpu69:2099071)WARNING: iscsi_vmk: iscsivmk_StartConnection:919: vmhba64:CH:2 T:0 CN:0: iSCSI connection is being marked "ONLINE"

• This indicates a connectivity issue between the ESXi host and iSCSI network. This is confirmed from the SCSI sense code "H:0x1" in the vmkernel logs: 

  YYYY-MM-DDT##:##:##.####  cpu30:2098300)NMP: nmp_ThrottleLogForDevice:3867: Cmd 0x12 (0x45baad9d0b88, 0) to dev "naa.##" on path "vmhba64:C2:T0:L0" Failed:
  YYYY-MM-DDT##:##:##.####  cpu30:2098300)NMP: nmp_ThrottleLogForDevice:3875: H:0x1 D:0x0 P:0x0 .

  Host Status

  [0x1]

  NO_CONNECT

  This status is returned if the connection is lost to the LUN. This can occur if the LUN is no longer visible to the host from the array side or if the physical connection to the array has been removed.

• Since the iSCSI LUNs are facing connectivity issues, the associated datastores will experience "Lost access to volume due to connectivity issues" alerts. This can also be seen in vobd logs:

  YYYY-MM-DDT##:##:##.#### : [scsiCorrelator] : [esx.problem.storage.connectivity.lost] Lost connectivity to storage device naa.##. Path vmhba64:C4:T0:L0 is down. Affected datastores: "###".
  YYYY-MM-DDT##:##:##.#### : [scsiCorrelator] : [vob.scsi.scsipath.pathstate.dead] scsiPath vmhba64:C4:T0:L1 changed state from on

Resolution

To resolve this issue:

• Verify the storage array configuration.

• Verify the host networking configuration.

  • To fix the VMkernel networking misconfiguration:

    1. When using multiple VMkernel ports for software iSCSI, ensure that the number of VMkernel ports is lesser than or equal to the number of physical network interfaces.

    2. Check MTU size across your environment and make it uniform (regular, Jumbo frames).

    3. Ensure following Best Practices for Configuring Networking with Software iSCSI

If the issue still persists, collect the TCP-dump during the time of the issue and share the same with the storage vendor to identify the reason for disconnection.

Additional Information

Handling Transient APD Conditions

iSCSI connection flapping between 'offline' and 'online'

Lost access to volume due to connectivity issues OR Path redundancy to storage device degraded