Storage adapter (vmhba) HBA card is showing as "offline" on ESXi Host.

Products

• VMware vSphere ESXi

Issue/Introduction

• From VC, multiple storage adapters or HBA cards are showing as Status Offline, as vmhba# in the example below:
  
  

• Or even a newly installed HBA card on ESXi host is showing on vCenter.
• One or more ESXi hosts are impacted.
• A power outage or any recent changes in the fabric layer may have been performed recently.
• The vmhbas may be online, but certain paths are offline or not seen. 
• Check the driver and firmware versions installed for the impacted adapter and if they are supported with the ESXi version installed in the host.

                      For reference on how to check the versions, please refer to Determining Network/Storage firmware and driver version in ESXi.

• A storage rescan was performed, but it didn't change the status of the adapter.
• The vmhba was reset from the ESXi with the command below, but the issue persists.
  • For FC:
    
    # esxcli storage san fc reset -A vmhba#
     
  • For FCoE:
    
    # esxcli storage san fcoe reset -A vmhba#

 

• From the var/run/log/vmkernel.log, the following events may be seen:

         [YYYY-MM-DDTHH:MM:SS] cpu49:2098184)WARNING: iodm: vmk_IodmEvent:194: vmhba6: FRAME DROP event has been observed 17 times in the last one minute. This suggests a problem with Fibre Channel link/switch!.

         [YYYY-MM-DDTHH:MM:SS] cpu51:2097288)ScsiDeviceIO: 4115: Cmd(0x45d993b3e748) 0x28, CmdSN 0x80000073 from world 3794523 to dev "naa.624a93703440e835806f4c8e000113fb" failed H:0x2 D:0x0 P:0x0

 

• A host reboot was also performed, but it didn't resolve the issue.
  
  
  
  

Environment

VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

• This is an indication of issues beyond ESXi regarding the storage adapter or the fabric layer.

 

• If the frame drop events are seen, please refer to the below:

FRAME DROP events suggest a possible connectivity issue or congestion. An erroneous FC Port can also lead to this issue.

 

• If the error "failed H:0x2 D:0x0 P:0x0" is seen, please refer to the below:

Host Status [H:0x2] :- This status is returned when the HBA driver is unable to issue a command to the device. This status occurs due to dropped FCP frames in the environment.

For [H:0x2] refer article: Understanding SCSI host-side NMP errors/conditions in ESXi

Resolution

Get the physical adapter checked and engage your fabric vendor to investigate and mitigate the issue.

Once the problem in the fabric layer is mitigated, the impacted hosts may need a reboot to refresh the information on the adapters.

 

If the adapter(s) and the fabric layer are validated as healthy by their vendors and the issue still persists, please open a ticket with VMware by Broadcom Technical Support.

Additional Information

Forcing a Fabric Login reset on Fibre Channel and FCOE Adapters

FRAME DROP event has been observed