Path Redundancy to Storage Device Degraded message on the ESXi host

Products

VMware Live Recovery VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

Of the different multi-pathing options that are available when interacting with a storage device, you might hit a scenario where one (or more than one) of these paths might not be available for you to leverage the entire depth of VMware multi-pathing.

At no point in time, even if the VMware connectivity to the back-end device is lost, other available paths can still be leveraged and communication with them should still continue, and there would not be any impact to the VMs or their functionality. However, there could still be a dip observed in the VMs overall performance.

The Task and Events tab of the ESXi host on the vSphere Client reports the error similar to:

Path redundancy to storage device naa.################################degraded. Path vmhba3:C0:T1:L7 is down. Affected datastores: Datastore1.

or

Lost access to volume ########-########-####-############# (Datastore1) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.

Environment

VMware vSphere ESXi 6.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Resolution

Examining this error:

Path redundancy to storage device naa.################################ degraded. Path vmhba3:C0:T1:L7 is down. Affected datastores: Datastore1.

The above error message indicates that of the different multi-pathing options in place to interact with device "naa.################################", only one path "vmhba3:C0:T1:L7" is down, and this path is no longer usable.

Breaking down the specifics of this path - vmhba3:C0:T1:L7, you can see several interaction points (or potential failure points):

vmhba3 - HBA (Host Bus Adapter)
C0 - Channel 0
T1 - Target 1 (Storage Processor or Storage Controller Port)
L7 - LUN 7 (Logical Unit Number or Disk Unit)

Of these, two of them could be rooted on the VMware stack (HBA/Channel), while the rest of them are external to VMware stack.

Majority of the time, this issue occurs due to some form of miscommunication taking place between the device driver and firmware versions of the PCI devices being used to interact with the storage devices.

Check HBA and NIC driver version and firmware version compatibility. For more information, see Determining Network/Storage firmware and driver version in ESXi (broadcom.com)

Note: For naa.################################

Network Address Authority Identifier

naa. - The Global Standard abbreviation can be NAA, T10, EUI etc...
# - Tells us this is IEEE registered extended format 6
######- Hexadecimal IEEE OUI
######### - Vendor-specific Identifier
################- Vendor-specific Identifier (extension)

NAA stands for Network Addressing Authority identifier. EUI stands for Extended Unique Identifier. The number is guaranteed to be unique to that LUN. The NAA or EUI identifier is the preferred method of identifying LUNs and the number is generated by the storage device. Since the NAA or EUI is unique to the LUN, if the LUN is presented the same way across all ESXi hosts, the NAA or EUI identifier remains the same. For more information on these standards, see the SPC-3 documentation from the International Committee for Information Technology Standards (T10).

For additional information, see Identifying disks when working with VMware ESXi (broadcom.com)

Disclaimer: VMware is not responsible for the reliability of any data, opinions, advice, or statements made on third-party websites. Inclusion of such links does not imply that VMware endorses, recommends, or accepts any responsibility for the content of such sites.

To investigate the actual failure or to eliminate possible issues

List all the paths for the device by running this command:

esxcfg-mpath -b -d device ID

For example:

esxcfg-mpath -b -d naa.################################

vmhba0:C0:T0:L11 LUN:11 state:active fc Adapter: WWNN: ##:##:##:##:##:##:##:## WWPN: ##:##:##:##:##:##:##:## Target: WWNN: ##:##:##:##:##:##:##:## WWPN: ##:##:##:##:##:##:##:##
vmhba0:C0:T1:L11 LUN:11 state:active fc Adapter: WWNN: ##:##:##:##:##:##:##:## WWPN: ##:##:##:##:##:##:##:## Target: WWNN: ##:##:##:##:##:##:##:## WWPN: ##:##:##:##:##:##:##:##
vmhba1:C0:T1:L11 LUN:11 state:active fc Adapter: WWNN: ##:##:##:##:##:##:##:## WWPN: ##:##:##:##:##:##:##:## Target: WWNN: ##:##:##:##:##:##:##:## WWPN: ##:##:##:##:##:##:##:##
vmhba1:C0:T0:L11 LUN:11 state:dead fc Adapter: Unavailable Target: Unavailable

In this example, you can see that of the four different multipathing options, only one is not active. It is important to understand that the remaining three paths are active and the connectivity to the back end device are not impacted. For more information, see Obtaining LUN pathing information for ESXi hosts (broadcom.com)

Important: A rescan is required after any change is made to see if the targets are detected.

Check that a rescan does not restore visibility to the targets. For more information on rescanning the storage device by using the command-line interface and VMware vSphere Client, see Performing a rescan of the storage on an ESXi host (broadcom.com)
Determine whether the connectivity issue is with the iSCSI storage or the fiber storage. Perform one of these options depending on what your connectivity issue is.

To troubleshoot the connectivity to the iSCSI storage using the software initiator

Check whether a ping to the storage array fails from the ESXi host. For more information, see Testing network connectivity with the ping command (broadcom.com)
Check whether a vmkping to each network portal of the storage array fails. For more information, see Testing VMkernel network connectivity with the vmkping command (broadcom.com)
Check that the initiator is registered on the array. Contact your storage vendor for instructions on this procedure.
Ensure that these physical hardware(s) are correctly functioning:

Ethernet Switch
Ethernet Cables between the ESXi Host and the Switch
Ethernet Cables between the Switch and the Array

To troubleshoot the connectivity to the FC storage

The fiber switch zoning configuration permits the ESX host to see the storage array. Consult your switch vendor if you require assistance.
The fiber switch propagates RSCN messages to the ESXi hosts.
Check the physical hardware, including:

The Storage Processors on the Array
The FC Switch and the Gigabit Interface Converter (GBIC) units in the switch
The FC cables between the Switch and the Array
The Array itself