Storage adapter (vmhba) HBA card is showing as "offline" on ESXi Host.

Products

• VMware vSphere ESXi

Issue/Introduction

• From VC, multiple storage adapters or HBA cards are showing as Status Offline, as vmhba# in the example below:
  
  

• Unable to see the newly installed HBA card on ESXi host its showing as offline.
• One or more ESXi hosts are impacted.
• This issue could be due to power outage or any recent changes in the fabric layer.
• Few vmhba's may be online, but certain paths are offline. 
• Check the driver and firmware versions installed for the impacted adapter and if they are supported with the ESXi version installed in the host.

                      For reference on how to check the versions, please refer to Determining Network/Storage firmware and driver version in ESXi.

• Storage rescan was performed, but it didn't change the status of the adapter.
• The vmhba was reset from the ESXi with the command below, but the issue persists.
  • For FC:
    
    # esxcli storage san fc reset -A vmhba#
     
  • For FCoE:
    
    # esxcli storage san fcoe reset -A vmhba#
• If it's is Cisco qlogic adapter, try to run the below command to check the HBA driver and firmware version
  •  /usr/lib/vmware/vmkmgmt_keyval/vmkmgmt_keyval -l -i vmhba#/qlogic

  • Sample Output:

    Listing keys:
ADAPTER
string

    Cisco QLE2742 Dual Port 32Gb FC to PCIe Gen3 x8 Adapter for QLE2742:
FC Firmware Version: 9.12.00 (d0d5), Driver version 5.3.81.2

    NVMe Host Device Name: vmhba$

    FC Host Information:
WWNN: ##:##:##:##:##:##:##:##
WWPN: ##:##:##:##:##:##:##:##
Port ID: 000000 NPortHandle: 0

    BIOS version 3.62
FCODE version 4.11
EFI version 7.27
Flash FW version 9.12.00
Multiboot Version 10.02.08 -> On Cisco KVM Console - Firmware version will be mentioned as 10.02.08

     

• On ESXi host SSH/Putty session HBA card state will be in "Link Down", when you run the below command: 
  • esxcli storage san fc list
    
    
  • Sample Output:
    
    Adapter: vmhba#
Port ID: 000000
Node Name: ##:##:##:##:##:##:##:##
Port Name: ##:##:##:##:##:##:##:##
Speed: 0 Gbps
Port Type: LPort
Port State: LINK DOWN
Model Description: Cisco QLE2742 Dual Port 32Gb FC to PCIe Gen3 x8 Adapter
Hardware Version:
OptionROM Version: 3.62
Firmware Version: 9.12.00 (d0d5)
Driver Name: qlnativefc
DriverVersion: 5.3.81.2

 

• From the vmkernel.log below events are seen.

         [YYYY-MM-DDTHH:MM:SS] cpu49:2098184)WARNING: iodm: vmk_IodmEvent:194: vmhba6: FRAME DROP event has been observed 17 times in the last one minute. This suggests a problem with Fibre Channel link/switch!.

         [YYYY-MM-DDTHH:MM:SS] cpu51:2097288)ScsiDeviceIO: 4115: Cmd(0x45d993b3e748) 0x28, CmdSN 0x80000073 from world 3794523 to dev "naa.624a93703440e835806f4c8e000113fb" failed H:0x2 D:0x0 P:0x0

 

• A host reboot also was performed but it didn't resolve the issue.
  
  
  
  

Environment

VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

This is an indication of issues beyond ESXi regarding the storage adapter or the fabric layer.

FRAME DROP events suggests a possible connectivity issue or congestion. An erroneous FC Port can also lead to this issue.

Host Status [H:0x2] :- This status is returned when the HBA driver is unable to issue a command to the device. This status occurs due to dropped FCP frames in the environment.

For [H:0x2] refer article: Understanding SCSI host-side NMP errors/conditions in ESXi

Resolution

Get the physical adapter checked and engage your fabric vendor to investigate and mitigate the issue.

Once the problem in the fabric layer is mitigated, the impacted hosts may need a reboot to refresh the information on the adapters.

 

If the adapter(s) and the fabric layer are validated as healthy by their vendors and the issue still persists, please open a ticket with VMware by Broadcom Technical Support.

Additional Information

Forcing a Fabric Login reset on Fibre Channel and FCOE Adapters

FRAME DROP event has been observed