Lost or degraded connectivity to storage device

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

 

             

Summary

 

    The event indicates a loss in connectivity to the specified storage device. The path indicated is the last path that went down.

 

 

    For example:

• Lost connectivity to storage device naa.################################. Path vmhba35:C1:T0:L7 is down. Affected datastores: "########"
• Path redundancy to storage device or Lost path redundancy to storage device naa.################################. Path vmhba35:C1:T0:L7 is down.

 

Impact

 

   Any virtual machines using the affected datastore may become unresponsive.

 

   If the service console root file system is on the affected datastore, then the service console becomes unresponsive and you cannot manage your ESX host.

Environment

VMware vCenter Server 8.x
VMware vSphere ESXi 8.x
VMware vCenter Server 7.x
VMware vSphere ESXi 7.x
VMware vCenter Server 6.x
VMware vSphere ESXi 6.x

Cause

The events occurs if there is loss of connection to Storage Devices, causing paths to fail or inaccessible.

Resolution

The storage device reading (vmhba35:C1:T0:L7) mentioned in the example earlier contains several potential failure points:

• vmhba35 - HBA (Host Bus Adapter)
• C1 - Channel
• T0 - Target (Storage processor port)
• L7 - LUN (Logical Unit Number or Disk Unit)

To determine the actual failure or to eliminate possible issues:

Note: All mentioned commands are available via vSphere Command-Line

1. Identify the available storage paths to the reported storage device by running the esxcfg-mpath -l command. For more information, see Obtaining LUN pathing information for ESX hosts.
2. Check that a rescan does not restore visibility to the targets. Follow the steps provided in Performing a re-scan of the storage to rescan the storage device by using the command-line interface and VMware vSphere Client.
3. Determine whether the connectivity issue is with the iSCSI storage or the fiber storage. Perform one of the following depending on what your connectivity issue is:
    
   • To troubleshoot the connectivity to the iSCSI storage using the software initiator:
      
     1. Check whether a ping to the storage array fails from ESX. For more information, see Testing network connectivity with the Ping command.
     2. Check whether a vmkping to each network portal of the storage array fails. For more information, see Testing VMkernel network connectivity with the vmkping command.
     3. Check that the initiator is registered on the array. Contact your storage vendor for instructions on this procedure.
     4. Check that the following physical hardware is correctly functioning:
         
        • Ethernet switch
        • Ethernet cables between the switch and the ESX host

        • Ethernet cables between the switch and the storage array

   • To troubleshoot the connectivity to the fiber attached storage, check the following:
     
      
     1. The fiber switch zoning configuration permits the ESX host to see the storage array. Consult your switch vendor if you require assistance.
     2. The fiber switch propagates RSCN messages to the ESX hosts. 
         
4. Check the physical hardware for the following:
    
   • The storage processors on the array.
   • The fiber switch and the Gigabit Interface Converter (GBIC) units in the switch.
   • The fiber cables between the fiber switch and the array.
   • The array itself.

Note: You must re-scan after any change is made to verify whether the targets are detected.

If the issue still persists please engage storage vendor, in order to validate,

1. LUN presentations and datastore connections.

2. Fiber Switch and Fiber Cables.

3. Fiber switch zoning.

4. Known issues on Storage that requires upgrade.