vCenter reports "Lost access to volume" events.

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms : 

• "Lost access to volume" events are displayed in vCenter.
  
• The /var/run/log/hostd.log file confirms the reported events:
  YYYY-MM-DDTHH:MM.SSSZ info hostd[2100792] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 13961 : Lost access to volume ########-########-####-############ (Datastore name) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.
YYYY-MM-DDTHH:MM.SSSZ info hostd[2110692] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 13962 : Successfully restored access to volume ########-########-####-############ (Datastore name) following connectivity issues.

Environment

VMware vSphere ESXi 6.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

Datastore heartbeat failures occurred due to I/O errors. Hostd reports "Lost access to volume" events If the datastore heartbeat does not recover within 18 seconds.

Refer : Understanding lost access to volume messages in ESXi

Cause validation:

        

• In var/run/log/hostd.log confirmed the lost access volume error at the same time issue reported in vcenter
  2025-09-29T15:26:02.780Z In(166) Hostd[2464849]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 10839 : Lost access to volume  ########-########-####-############ Datastore name due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.
  2025-09-29T15:26:05.383Z In(166) Hostd[2099436]: [Originator@6876 sub=Hostsvc.VmkVprobSource] VmkVprobSource::Post event: (vim.event.EventEx) 

• var/run/log/vobd.log file indicates heartbeat timeout events for the datastore.
  YYYY-MM-DDTHH:MM.SSSZ: [vmfsCorrelator] 28263874625980us: [vob.vmfs.heartbeat.timedout] ########-########-####-############ Datastore name
YYYY-MM-DDTHH:MM.SSSZ: [vmfsCorrelator] 28264136792700us: [esx.problem.vmfs.heartbeat.timedout] ########-########-####-############ Datastore name
YYYY-MM-DDTHH:MM.SSSZ: [vmfsCorrelator] 28263915142571us: [vob.vmfs.heartbeat.recovered] Reclaimed heartbeat for volume ########-########-####-############ (Datastore name): [Timeout] [HB state abcdef02 offset 3440640 gen 1695 stampUS 28263915141361 uuid ########-########-####-############ jrnl <FB 8388610> drv 24.82]
YYYY-MM-DDTHH:MM.SSSZ: [vmfsCorrelator] 28264177309659us: [esx.problem.vmfs.heartbeat.recovered] ########-########-####-############ Datastore name
  
  
• var/run/log/vmkernel.log file reports I/O (0x89 : Compare and Write, 0x28 : Read, 0x12 : Inquiry) failures with scsi codes "H:0x2 and H0x5".
  
  H:0x2 will be reported when HBA driver is unable to issue a command to the device. This status can occur due to dropped FCP frames in the environment.
  H:0x5 indicates that the driver has to abort commands in-flight to the target. This can occur due to a command timeout or parity error in the frame. 
  
  

  YYYY-MM-DDTHH:MM.SSSZ cpu1:2097758)NMP: nmp_ThrottleLogForDevice:3867: Cmd 0x89 (0x45b8f2dd8988, 2097177) to dev "naa.########################" on path "vmhba0:C0:T0:L0" Failed:
YYYY-MM-DDTHH:MM.SSSZ cpu1:2097758)NMP: nmp_ThrottleLogForDevice:3875: H:0x5 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x430562599500 CmdSN 0x1283b8e
YYYY-MM-DDTHH:MM.SSSZ cpu1:2097758)ScsiDeviceIO: 4154: Cmd(0x45b8f2dd8988) 0x89, cmdId.initiator=0x430562599500 CmdSN 0x1283b8e from world 2097177 to dev "naa.########################" failed H:0x5 D:0x0 P:0x0 Cancelled from driver layer
YYYY-MM-DDTHH:MM.SSSZ cpu0:2097760)NMP: nmp_ThrottleLogForDevice:3867: Cmd 0x28 (0x45b8f2d0af88, 0) to dev "naa.########################" on path "vmhba0:C0:T0:L0" Failed:
YYYY-MM-DDTHH:MM.SSSZ cpu0:2097760)NMP: nmp_ThrottleLogForDevice:3875: H:0x2 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x430562599500 CmdSN 0x1283b90
YYYY-MM-DDTHH:MM.SSSZ cpu0:95783404)NMP: nmp_ThrottleLogForDevice:3867: Cmd 0x12 (0x45d8e6ee1548, 0) to dev "naa.########################" on path "vmhba0:C0:T0:L0" Failed:
YYYY-MM-DDTHH:MM.SSSZ cpu0:95783404)NMP: nmp_ThrottleLogForDevice:3875: H:0x5 D:0x0 P:0x0 . Act:NONE. cmdId.initiator=0x453893a1bb38 CmdSN 0x0

Resolution

• Involve switch and storage vendor to investigate and assist with the IO failures.
• Validate the driver/Firmware compatibility using Broadcom Compatibility Guide