Lost connectivity or Slowness when accessing local VMFS datastore due to issued Storage Controller (vmhba) Aborts

Products

• VMware vSphere ESXi

Issue/Introduction

The issue is with a Storage Controller (vmhba) connecting a local VMFS Datastore on an ESXi Host.

 

You might experience one or more of the following Issues:

• Storage vMotion(s) are very slow or failing
• VM(s) located on the same VMFS local Datastore are all slow or reporting issues (incl. VM is not getting powered on )
• Slowness on the ESXi Host owning the affected local VMFS Datastore where all affected VM(s) are located

• On the ESXi Host connecting to the affected local VMFS Datastore, Heartbeat timeouts to it are observed

Example via /var/log/vobd.log: 

[vmfsCorrelator] ##########: [vob.vmfs.heartbeat.timedout] <VMFS datastore UUID> <VMFS datastore name>
[vmfsCorrelator] ##########: [esx.problem.vmfs.heartbeat.timedout] <VMFS datastore UUID> <VMFS datastore name>

 

• On the ESXi Host connecting to the affected local VMFS Datastore, you receive an I/O error when attempting to access the Host Logs

Example related to vmkernel.log

 

 

At the time of the occurrence of one or more of the issues listed above, the ESXi Host logs do show Aborts being initiated by the Host and/or the VM Guest OS itself

Example via /var/log/vmkernel.log:

YYYY-MM-DDTHH:MM:SS.ZZ In(182) vmkernel: cpu8:2097572)lsi_mr3_######: mfi_TaskMgmt: 689: Processing taskMgmt virt reset for device: vmhba#:C#:T#:L#

YYYY-MM-DDTHH:MM:SS.ZZ In(182) vmkernel: cpu47:2167508)lsi_mr3_######: mfi_TaskMgmt: 689: Processing taskMgmt abort for device: vmhba#:C#:T#:L#

YYYY-MM-DDTHH:MM:SS.ZZ Wa(180) vmkwarning: cpu26:2098293)WARNING: HPP: HppScsiThrottleLogForDevice:585: Cmd 0x28 (0x45b91aa82b40, 2100022) to dev "#############################" on path "vmhba#:C#:T#:L#" Failed:

 

 

Environment

vSphere ESXi (All Versions)

Cause

Issue with Storage Controller (vmhba) and/or its installed Firmware.

 

Resolution

Confirm that installed Controller (vmhba) Driver and Firmware are supported by locating the related Controller listing on the Broadcom Compatibility Guide.

Contact your HW Vendor further investigation.

 

 

Additional Information

Supplemental Documentation:

• Host-Level Hardware Compatibility Checks
• Confirming ESX/ESXi host hardware (System, Storage, and I/O) compatibility

 

Commands to determine further details (to be run on the affected ESXi Host):

• Storage Controller (vmhba): Model, Driver and Firmware:

esxcli storage san sas list

 

Example:

 

[root@#########] esxcli storage san sas list

   Device Name: vmhba#

   SAS Address: ############

   Physical ID: #

   Minimum Link Rate: 0 Mbps

   Maximum Link Rate: 0 Mbps

   Negotiated Link Rate: 0 Mbps

   Model Description: ############

   Hardware Version: #

   OptionROM Version: ############

   Firmware Version: ############

   Driver Name: lsi_mr3

   Driver Version: ############

 

 

• Relationship between affected local VMFS Datastore and associated local Disk it resides on:

localcli storage vmfs extent list

Example:

 

 

 

• Relationship between Storage Controller (vmhba) and local Disks:

esxcfg-scsidevs -A

 

Example: