Error : "Lost access to volume" reports on vCenter.

Products

• VMware vSphere ESXi

Issue/Introduction

• ESXi hosts temporarily lose storage paths, causing virtual machines (VMs) to lose disk connectivity for several minutes before automatically recovering.
• The vCenter server reports "Lost access to volume" events.
  
  
• The /var/run/log/hostd.log file confirms the reported events:
  
  YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2098819]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 31356 : Lost access to volume ########-########-####-############ (####-######-#######-####-########) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.
  
  
• The /var/run/log/vobd.log file indicates a datastore heartbeat timeout:
  
  YYYY-MM-DDTHH:MM.SSSZ In(14) vobd[2097766]:  [vmfsCorrelator] 8008405815191us: [esx.problem.vmfs.heartbeat.timedout] ########-########-####-############ ####-######-#######-####-########.
  
  
• The /var/run/log/vmkernel.log file shows I/O aborts on the nfnic driver:
  
  
  YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu1:2098622)Fil6: 4289: '####-######-#######-####-########': Fil6 file IO (<FD c35 r1>) : IO was aborted by VMFS via a virt-reset on the device
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu1:2098622)Fil6: 4251: ioCtx: 0x45bac76abcc0, world: 2098622, overallStatus: IO was aborted by VMFS via a virt-reset on the device, token: 0x0, tokenStatus: N, txnToken: 0x0, txnTokenStatus: N, totalIOSize: 512, maxIOLength: 1$
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu44:2097268)ScsiDeviceIO: 4605: Cmd(0x45dac63e0d80) 0x8a, cmdId.initiator=0x430fbdc12440 CmdSN 0xffffea01bba218d0 from world 4895060 to dev "naa.################################" failed H:0x8 D:0x0 P:0x0 Cancelled from
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu44:2097268)device layer
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu34:2098095)nfnic: <1>: INFO: fnic_abort_cmd: 3810: Abort cmd called for Tag: 0x502  issued time: 10310 ms CMD_STATE: FNIC_IOREQ_CMD_PENDING CDB Opcode: 0x8a  sc:0x45dac6270980 flags: 0x3 lun: 1 target: 0x4c0100
YYYY-MM-DDTHH:MM.SSSZ Wa(180) vmkwarning: cpu34:2098095)WARNING: nfnic: <1>: fnic_abort_cmd: 3825: Abort for cmd tag: 0x502 in pending state
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu44:2097268)ScsiDeviceIO: 4605: Cmd(0x45dac620a980) 0x8a, cmdId.initiator=0x430fbdc12440 CmdSN 0xffffea01bc216400 from world 4895060 to dev "naa.################################" failed H:0x8 D:0x0 P:0x0 Cancelled from
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu44:2097268)device layer
  2026-07-12T21:52:19.620Z In(182) vmkernel: cpu34:2098095)nfnic: <1>: INFO: fnic_abort_cmd: 3810: Abort cmd called for Tag: 0x50d  issued time: 10012 ms CMD_STATE: FNIC_IOREQ_ABTS_PENDING CDB Opcode: 0x8a  sc:0x45dac62a0380 flags: 0x43 lun: 1 target: 0x4c0060
YYYY-MM-DDTHH:MM.SSSZ Wa(180) vmkwarning: cpu34:2098095)WARNING: nfnic: <1>: fnic_abort_cmd: 3821: Abort for cmd tag: 0x50d already issued
  
  
• The NFNICs adapters are used to configure storage paths.
  
  esxcli storage core adapter list
  
  HBA Name  Driver    Link State  UID                                   Capabilities         Description
--------  --------  ----------  ------------------------------------  -------------------  -----------
vmhba0    nfnic     link-up     fc.################:################  Second Level Lun ID  (####:##:##.#) Cisco Corporation Cisco UCS VIC Fnic Controller
vmhba2    nfnic     link-up     fc.################:################  Second Level Lun ID  (####:##:##.##) Cisco Corporation Cisco UCS VIC Fnic Controller

Environment

• VMware vSphere ESXi 7.x
• VMware vSphere ESXi 8.x
• VMware vSphere ESXi 9.x

Cause

• The installed Cisco nfnic driver version is incompatible with the running firmware version.

• Checking driver and firmware combinations confirms that the driver used is not compatible with the firmware version installed:

   
  Example : The nfnic driver version "5.0.0.43" is paired with firmware version "4.4(3c)".
  
  esxcli storage san fc list

FcDevice:
   Adapter: vmhba0
   Port ID: 4C0146
   Node Name: ##:##:##:##:##:##:##:##
   Port Name: ##:##:##:##:##:##:##:##
   Speed: 16 Gbps
   Port Type: NPort
   Port State: ONLINE
   Model Description: #######
   Hardware Version: ##
   OptionROM Version: ###########
   Firmware Version: 4.4(3c)
   Driver Name: nfnic
   DriverVersion: 5.0.0.43
  
  According to the Broadcom compatibility guide, driver version 5.0.0.43 should be paired with firmware version 4.5(3)

Resolution

1. Engage hardware vendor and update the nfnic driver firmware to the minimum required version as per the Broadcom compatibility guide. 

Additional Information

KB 381648: Cisco UCS servers log FNIC aborts in ESXi