Path Redundancy to Storage Device Degraded message on the ESXi host
search cancel

Path Redundancy to Storage Device Degraded message on the ESXi host

book

Article ID: 318935

calendar_today

Updated On:

Products

VMware Live Recovery VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

Of the different multi-pathing options that are available when interacting with a storage device, you might hit a scenario where one (or more than one) of these paths might not be available for you to leverage the entire depth of VMware multi-pathing. 

At no point in time, even if the VMware connectivity to the back-end device is lost, other available paths can still be leveraged and communication with them should still continue, and there would not be any impact to the VMs or their functionality. However, there could still be a dip observed in the VMs overall performance.
  • The Task and Events tab of the ESXi host on the vSphere Client reports the error similar to:

    Path redundancy to storage device naa.################################degraded. Path vmhba3:C0:T1:L7 is down. Affected datastores: Datastore1.

    or 

    Lost access to volume ########-########-####-############# (Datastore1) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.
  • In case the issue is caused from the fabric/switch layer then you may find below log entries in vmkernel logs:
[YYYY-MM-DDTHH:MM:SS] cpu79:2098718)lpfc: lpfc_handle_status:5637: 3:(0):3271: FCP cmd x28 failed <0/24> sid x010800, did x010600, oxid x23bd iotag xee3 Local Reason x1d Returning Host Busy
[YYYY-MM-DDTHH:MM:SS] cpu79:2098718)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.################################" state in doubt; requested fast path state update...
[YYYY-MM-DDTHH:MM:SS] cpu77:10146015)StorageDevice: 7059: End path evaluation for device naa.################################
[YYYY-MM-DDTHH:MM:SS] cpu41:5273215)lpfc: lpfc_handle_status:5637: 3:(0):3271: FCP cmd x2a failed <0/24> sid x010800, did x010600, oxid x1bf7 iotag x71d Time Out Returning Host Busy
[YYYY-MM-DDTHH:MM:SS] cpu0:2098717)NMP: nmp_ThrottleLogForDevice:3798: last error status from device naa.################################ repeated 10 times
[YYYY-MM-DDTHH:MM:SS] cpu27:2097317)NMP: nmp_ThrottleLogForDevice:3875: H:0x2 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x430df90129c0 CmdSN 0x80000049
 
         [YYYY-MM-DDTHH:MM:SS] cpu37:35979576)qlnativefc: vmhba64(86:0.0): qlnativefcStatusEntry:2038:(2:0) Port down status: port-state=0x3
         [YYYY-MM-DDTHH:MM:SS] cpu14:2097637)qlnativefc: vmhba64(86:0.0): qlnativefcStatusEntry:2038:(2:0) Port down status: port-state=0x3
         [YYYY-MM-DDTHH:MM:SS] cpu69:2097221)qlnativefc: vmhba64(86:0.0): qlnativefcStatusEntry:2038:(2:0) Port down status: port-state=0x3
         [YYYY-MM-DDTHH:MM:SS] cpu27:74120652)qlnativefc: vmhba64(86:0.0): qlnativefcStatusEntry:2038:(2:1) Port down status: port-state=0x3
  • These messages are generated by the Emulex LightPulse Fibre Channel (lpfc) driver on the ESXi host and indicate a device loss timeout—meaning the host failed to communicate with the storage target (WWPN ##:##.##:##:##:##:##:##) within the allowed timeout period:

2025-06-16T06:29:07.068Z cpu21:2098653)WARNING: lpfc: lpfc_dev_loss_tmo_handler:543: 1:(0):0203 Devloss timeout on WWPN  ##:##.##:##:##:##:##:## NPort x800301 Data: x8 x8 x3

  • The SCSI INQUIRY command to the device failed, indicating that ESXi is unable to reach or communicate with the LUN using the path vmhba3:C0:T0:L40.

2025-06-16T06:29:07.070Z cpu40:2098699)NMP: nmp_ThrottleLogForDevice:3867: Cmd 0x12 (0x45ba6c618b48, 0) to dev "naa.#############################" on path "vmhba3:C0:T0:L40" Failed:
2025-06-16T06:29:07.070Z cpu40:2098699)NMP: nmp_ThrottleLogForDevice:3875: H:0x1 D:0x0 P:0x0 . Act:NONE. cmdId.initiator=0x453a0d41bab8 CmdSN 0x0

Environment

VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Resolution

Examining this error:

Path redundancy to storage device naa.################################ degraded. Path vmhba3:C0:T1:L7 is down. Affected datastores: Datastore1.

The above error message indicates that of the different multi-pathing options in place to interact with device "naa.################################", only one path "vmhba3:C0:T1:L7" is down, and this path is no longer usable. 

Breaking down the specifics of this path - vmhba3:C0:T1:L7, you can see several interaction points (or potential failure points):

  • vmhba3 - HBA (Host Bus Adapter)
  • C0 - Channel 0
  • T1 - Target 1 (Storage Processor or Storage Controller Port)
  • L7 - LUN 7 (Logical Unit Number or Disk Unit)

Of these, two of them could be rooted on the VMware stack (HBA/Channel), while the rest of them are external to VMware stack. 

Majority of the time, this issue occurs due to some form of miscommunication taking place between the device driver and firmware versions of the PCI devices being used to interact with the storage devices. 

Check HBA and NIC driver version and firmware version compatibility. For more information, see Determining Network/Storage firmware and driver version in ESXi

Note: For naa.################################

Network Address Authority Identifier

naa. - The Global Standard abbreviation can be NAA, T10, EUI etc...
# - Tells us this is IEEE registered extended format 6
######- Hexadecimal IEEE OUI
######### - Vendor-specific Identifier
################- Vendor-specific Identifier (extension)

NAA stands for Network Addressing Authority identifier. EUI stands for Extended Unique Identifier. The number is guaranteed to be unique to that LUN. The NAA or EUI identifier is the preferred method of identifying LUNs and the number is generated by the storage device. Since the NAA or EUI is unique to the LUN, if the LUN is presented the same way across all ESXi hosts, the NAA or EUI identifier remains the same. For more information on these standards, see the SPC-3 documentation from the International Committee for Information Technology Standards (T10). 

For additional information, see Identifying disks when working with VMware ESXi

Disclaimer: VMware is not responsible for the reliability of any data, opinions, advice, or statements made on third-party websites. Inclusion of such links does not imply that VMware endorses, recommends, or accepts any responsibility for the content of such sites.

To investigate the actual failure or to eliminate possible issues

List all the paths for the device by running this command:

esxcfg-mpath -b -d device ID 

For example:

esxcfg-mpath -b -d naa.################################

vmhba0:C0:T0:L11 LUN:11 state:active fc Adapter: WWNN: ##:##:##:##:##:##:##:## WWPN: ##:##:##:##:##:##:##:##  Target: WWNN: ##:##:##:##:##:##:##:## WWPN: ##:##:##:##:##:##:##:##
vmhba0:C0:T1:L11 LUN:11 state:active fc Adapter: WWNN: ##:##:##:##:##:##:##:## WWPN: ##:##:##:##:##:##:##:##  Target: WWNN: ##:##:##:##:##:##:##:## WWPN: ##:##:##:##:##:##:##:##
vmhba1:C0:T1:L11 LUN:11 state:active fc Adapter: WWNN: ##:##:##:##:##:##:##:## WWPN: ##:##:##:##:##:##:##:##  Target: WWNN: ##:##:##:##:##:##:##:## WWPN: ##:##:##:##:##:##:##:##
vmhba1:C0:T0:L11 LUN:11 state:dead fc Adapter: Unavailable Target: Unavailable

In this example, you can see that of the four different multipathing options, only one is not active. It is important to understand that the remaining three paths are active and the connectivity to the back end device are not impacted. For more information, see Obtaining LUN pathing information for ESXi hosts

Important: A rescan is required after any change is made to see if the targets are detected.

  1. Check that a rescan does not restore visibility to the targets. For more information on rescanning the storage device by using the command-line interface and VMware vSphere Client, see Performing a rescan of the storage on an ESXi host
     
  2. Determine whether the connectivity issue is with the iSCSI storage or the fiber storage. Perform one of these options depending on what your connectivity issue is. 

To troubleshoot the connectivity to the iSCSI storage using the software initiator

  1. Check whether a ping to the storage array fails from the ESXi host. For more information, see Testing network connectivity with the ping command
     
  2. Check whether a vmkping to each network portal of the storage array fails. For more information, see Testing VMkernel network connectivity with the vmkping command
     
  3. Check that the initiator is registered on the array. Contact your storage vendor for instructions on this procedure.
     
  4. Ensure that these physical hardware(s) are correctly functioning:
  • Ethernet Switch
  • Ethernet Cables between the ESXi Host and the Switch
  • Ethernet Cables between the Switch and the Array

To troubleshoot the connectivity to the FC storage

  1. The fiber switch zoning configuration permits the ESX host to see the storage array. Consult your switch vendor if you require assistance.
     
  2. The fiber switch propagates RSCN messages to the ESXi hosts. 
     
  3. Check the physical hardware, including:
  • The Storage Processors on the Array
  • The FC Switch and the Gigabit Interface Converter (GBIC) units in the switch
  • The FC cables between the Switch and the Array
  • The Array itself
Powered by Wolken Software