ESXi host loses storage access intermittently with the error "Lost access to volume"

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

 

• The ESXi host loses storage access intermittently.

• In the /var/log/hostd.log file, you see entries similar to:
  
  <YYYY-MM-DD>T<time> [4F1E1B70 info 'Vimsvc.ha-eventmgr'] Event 205 : Lost access to volume XXXXXXXX-XXXXXXXX-XXXX-XXXXXXXXXXXX due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.
  </time><YYYY-MM-DD>T<time> [4F480B70 info 'Vimsvc.ha-eventmgr'] Event 210 : Successfully restored access to volume XXXXXXXX-XXXXXXXX-XXXX-XXXXXXXXXXXX  following connectivity issues.
  
  </time>
• In the /var/log/vmkernel.log file, you see entries similar to:
  
  <YYYY-MM-DD>T<time> cpuXX:XXXXXX)WARNING: LinScsi: SCSILinuxAbortCommands:1891: Failed, Driver fnic, for vmhba1</time>
<YYYY-MM-DD>T<time> cpuXX:XXXXXX)<7>fnic : 2 :: Abort Cmd called FCID 0x640380, LUN 0xXX TAG bd flags 3</time>
<YYYY-MM-DD>T<time> cpuXX:XXXXXX)<6>fnic : 2 :: abort reject recd. id 189</time>
<YYYY-MM-DD>T<time> cpuXX:XXXXXX)<7>fnic : 2 :: abts cmpl recd. id 189 status FCPIO_ITMF_REJECTED</time>
<YYYY-MM-DD>T<time> cpuXX:XXXXXX)<7>fnic : 2 :: Returning from abort cmd type 2 FAILED</time>
<YYYY-MM-DD>T<time> cpuXX:XXXXXX)WARNING: LinScsi: SCSILinuxAbortCommands:1891: Failed, Driver fnic, for vmhbaX</time>
<YYYY-MM-DD>T<time> cpuXX:XXXXXX)<7>fnic : 2 :: Abort Cmd called FCID 0x640380, LUN 0x65 TAG bf flags 3</time>
<YYYY-MM-DD>T<time> cpuXX:XXXXXX)<6>fnic : 2 :: abort reject recd. id 191</time>
<YYYY-MM-DD>T<time> cpuXX:XXXXXX)<7>fnic : 2 :: abts cmpl recd. id 191 status FCPIO_ITMF_REJECTED</time>
<YYYY-MM-DD>T<time> cpuXX:XXXXXX)<7>fnic : 2 :: Returning from abort cmd type 2 FAILED</time>

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vSphere ESXi 6.0
VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.5
VMware vSphere ESXi 7.0
VMware vSphere ESXi 8.0 

Resolution

To resolve this issue, driver and firmware must be inline and compatible:

 

1. Verify and update the driver and firmware to compatible versions for your Cisco Systems Inc Cisco VIC FCoE HBA.

   To check the HCL for Cisco FCoE CNAs, please see VMware Compatibility Guide for FCoE CNAs.
    
2. Update BIOS on the host to the latest available version.
    
3. Verify and update the driver and firmware to compatible versions for your Cisco Systems Inc Cisco VIC Ethernet NIC.
   
   To check the HCL for Cisco etheret NIcs, please see VMware Compatibility Guide for Cisco Network devices.