Error: Cannot connect to storage alarm triggers in vCenter

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms : 

• Frequent "Path redundancy to storage device degraded" Events are seen on vCenter.
• ESXi host may go unresponsive.
• Virtual machines enter a hung or frozen state, and standard operations, such as power on or power off, queue up and fail to execute.
• The alarm "Cannot connect to storage" triggers on the ESXi host.
  
  • This alarm triggers if any one of the below mentioned criteria is met.
    IF Lost Storage Connectivity
OR Lost Storage Path Redundancy
OR Degraded Storage Path Redundancy
OR Lost connection to NFS server
THEN trigger the alarm as unknown
    Note: These criteria apply to default alarm rules. To review current definitions, navigate to the affected ESXi host > Configure > Alarm Definitions.

• In the /var/run/log/vobd.log file, similar entries are seen:
  YYYY-MM-DDTHH:MM.SSSZ In(14) vobd[2098149]:  [scsiCorrelator] 6182941357us: [esx.problem.storage.redundancy.degraded] Path redundancy to storage device eui.################################ degraded. Path vmhba6:C0:T19:L0 is down. Affected datastores: Unknown.
YYYY-MM-DDTHH:MM.SSSZ In(14) vobd[2098149]:  [scsiCorrelator] 6182942941us: [esx.problem.storage.redundancy.degraded] Path redundancy to storage device eui.################################ degraded. Path vmhba6:C0:T19:L1 is down. Affected datastores: Unknown.
YYYY-MM-DDTHH:MM.SSSZ In(14) vobd[2098149]:  [scsiCorrelator] 6182944535us: [esx.problem.storage.redundancy.degraded] Path redundancy to storage device eui.################################ degraded. Path vmhba6:C0:T19:L2 is down. Affected datastores: Unknown.
  
  
• The /var/run/log/hostd.log file confirm that the VM power off operation gets stuck and queued up.
  2025-06-06T10:05:29.568Z Db(167) Hostd[2102696]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/########-########-####-############/VM/VM.vmx opID=########-########-auto-#####-h5:########-32-ff-#### sid=######## user=vpxuser:VSPHERE.LOCAL\Administrator] PowerOff request queued
2025-06-06T10:05:29.570Z In(166) Hostd[2102688]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/########-########-####-############/VM/VM.vmx] Upgrade is required for virtual machine, version: 15
2025-06-06T10:05:36.903Z Wa(164) Hostd[2102684]: [Originator@6876 sub=IoTracker] In thread 2102680, fopen("/vmfs/volumes/########-########-####-############/VM/VM.vmx") took over 392 sec.

Environment

• VMware vSphere ESXi 7.x
• VMware vSphere ESXi 8.x
• VMware vSphere ESXi 9.x

Cause

• The vmhba link going down frequently due to physical layer issue.

• The /var/run/log/vmkernel.log file indicates that the vhmba link down events.
  YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu93:2101535)qlnativefc: vmhba6(99:0.1): qlnativefcAsyncEvent:847:NVMe report link down
YYYY-MM-DDTHH:MM.SSSZIn(182) vmkernel: cpu93:2107156)qlnativefc: vmhba6(99:0.1): qlnativefcAsyncEvent:760:NVMe report link up.
  
  
• The command "esxcli storage san fc events get" confirm vmhba status link down events frequently.
  
  esxcli storage san fc events get
FC Event Log
  ------------
  YYYY-MM-DD HH:MM:SS.ms [vmhba6] LINK DOWN
YYYY-MM-DD HH:MM:SS.ms [vmhba6] LINK UP
YYYY-MM-DD HH:MM:SS.ms [vmhba6] LINK DOWN
YYYY-MM-DD HH:MM:SS.ms [vmhba6] LINK UP
YYYY-MM-DD HH:MM:SS.ms [vmhba6] LINK DOWN
YYYY-MM-DD HH:MM:SS.ms [vmhba6] LINK UP
  
  From the above example, it is confirmed that the VMHBA6 Link status is flapping, resulting in the paths going down intermittently.
  
  
• Verify the link failure count with the below mentioned command:
  
  "esxcli storage san fc stats get" 
FcStat:
   Adapter: vmhba6
   Tx Frames: 10447376
   Rx Frames: 20986931
   Lip Count: 0
   Error Frames: 0
   Dumped Frames: 0
   Link Failure Count: 168088
   Loss of Signal Count: 0
   PrimSeq Protocol Err Count: 0
   Invalid Tx Word Count: 0
   Invalid CRC Count: 0
   Input Requests: 6024784
   Output Requests: 30838
   Control Requests: 336718




Resolution

1. Engage the switch vendor to validate the health of the affected adapters and the physical switch ports.