Lost connectivity to local VMFS datastore due to aborts and slow completion of aborts

Products

• VMware vSphere ESXi

Issue/Introduction

ESXi reports "Lost access to volume <volume> due to connectivity issues" for a local datastore 

vobd.log reports this is due to datastore heartbeat timeout, e.g.

[vmfsCorrelator] 10283981076650us: [vob.vmfs.heartbeat.timedout] <VMFS datastore UUID> <VMFS datastore name>
[vmfsCorrelator] 10284129528213us: [esx.problem.vmfs.heartbeat.timedout] <VMFS datastore UUID> <VMFS datastore name>

At this time, vmkernel logs reports Task Management aborts (i.e. aborts originating from a higher layer such as VM guestOS , which driver processes but does not originate). Aborts take several seconds to complete once issued by the driver, for example:

lsi_mr3: mfi_TaskMgmt:690: Processing taskMgmt abort for device: vmhba#:C#:T#:L#
lsi_mr3: mfi_TaskMgmt:724: ABORT request for SN 15465642 Wld 2097217
lsi_mr3: mfi_TaskMgmt:739: TA : SMID 1476 IN 10283972s OUT 0s TM 10283980sDelta 8s

Environment

7.x
8.x

Cause

The logging indicates a firmware or hardware issue. 

Resolution

Confirm that driver/firmware for the local controller is supported. See Broadcom Compatibility Guide.

Engage hardware vendor for further investigation.