Storage paths are missing on multiple ESXI hosts
search cancel

Storage paths are missing on multiple ESXI hosts

book

Article ID: 391150

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

  • Storage paths to LUN are missing and some reported to be "Dead".
  • Storage paths are inconsistent.
  • /var/log/vmkernel.log reports "No connection" on paths
  • datastores may report to be inaccessible in the vSphere Client (if all paths impacted)

Validation:

1. Storage targets missing and target count may not match between the vmhba adapters on vCenter UI under Inventory>Cluster>Host>Storage adapters

 

 

2.Storage path to LUN reports to be "Dead" on vSphere UI under Inventory>Cluster>Host>Storage devices

 

 

3.Storage paths are inconsistent. You may observe this under vCenter UI under Inventory>Cluster>Host>Storage adapters

 

 

3.Lun disconnect with sense code H:0x1 with observed on /var/run/log/vmkernel.log

2025-03-02T17:06:59.799Z cpu92:2098345)NMP: nmp_ThrottleLogForDevice:3875: H:0x1 D:0x0 P:0x0 . Act:NONE. cmdId.initiator=0x453a1521b5f8 CmdSN 0x0

 

4.Cluster level storage rescan on vSphere UI >Inventory>Cluster>right-click may not fix the path issues.

        or

       Rescan from Host cli esxcli storage core adapter rescan --all 

 

 

Environment

VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

This issue is caused due to below activities done on storage array which leads to path issues or storage path to LUN state to be "Dead"

1.Reboot of Storage controller due to:
  • Firmware upgrade
  • To clear an erroneous state like target port etc.
  • For other maintenance reasons.
2.Storage controllers may not be load balanced. One of the controller may be heavily utilized.

If the connected array port has not been responsive for longer than the device timeout for the HBA driver, the HBA driver marks that device as missing. This causes the HBA driver to send a host status of NO_CONNECT (H:0x1) back to the ESXi/ESX storage stack layer, which in turn causes immediate failover to another good path (MRU/Fixed) or the removal of that path from the working paths list that is available for the Round Robin PSP to utilize. This affects paths to all LUNs provided through the array controller.

 

Resolution

To fix the path issue, Forcing a LIP (Loop Initialization Primitive) on the affected HBA should be able to bring back the connectivity without rebooting the Host

From an SSH session or command line, interface to the ESXi host:

esxcli storage san fc reset -A vmhbaX

Where vmhbaX is the adapter on which you are performing the reset.


In some cases, if a LIP reset fails to recover paths, a host reboot may resolve the issue. 

Additional Information

Loop Initialization Protocol (LIP), scans the interconnect, and causes the SCSI layer to be updated to reflect the devices currently on the bus. Essentially, an LIP is a bus reset, and causes device addition and removal with paths getting updated.

Powered by Wolken Software