Error: vob.vmfs.heartbeat.timedout during LACP uplink restoration on ESXi

Products

• VMware vSphere ESXi

Issue/Introduction

ESXi hosts lose access to iSCSI datastores for a brief period when an uplink in a Link Aggregation Group (LAG) returns to an online state. This occurs even if other uplinks in the LAG remain active.

Symptoms:

• iSCSI datastores report 'Lost access to volume' errors.
• Log entries in vobd.log show heartbeat timeouts shortly after an interface rejoins the LACP group:
  Shortly after the interface rejoined the LACP group, storage heartbeats began timing out:  [vob.vmfs.heartbeat.timedout] ####-####-####-#### 
  Heartbeats successfully recovered approximately 20 seconds later: [esx.problem.vmfs.heartbeat.timedout] ####-####-####-####
• Connectivity typically recovers automatically within seconds once the physical switch begins forwarding traffic.

Environment

• VMware vSphere ESXi 8.x
• iSCSI Storage with LACP (Link Aggregation Control Protocol) configured on the Distributed Virtual Switch (VDS).

Cause

• The brief loss of storage connectivity is caused by the upstream physical switch port not being fully ready to forward traffic immediately after the physical link state returns to "up".
• When the vmnic returns to an online state, the ESXi host re-adds it to the LACP port channel.
• The iSCSI vmkernel adapter begins utilizing the restored path based on the LAG hashing algorithm.
• If the physical switch port is in a 'learning' or 'initialization' phase and not yet forwarding data, the storage traffic is blackholed, leading to heartbeat timeouts.

Resolution

Engage with network team to investigate why physical switch failed to forward network traffic during the boot up.

Additional Information

• For steps to download latest releases, see Download Broadcom Products and Software.
• To speak with a customer representative or a Support Engineer see Contact Support. Scroll to the bottom of the page and click on your respective region.