Error: VMFS heartbeat timed out during large file transfers residing on local storage

Products

• VMware vSphere ESXi

Issue/Introduction

Performance degradation occurs when transferring large image files (OVF/ISO) to local datastores on ESXi 8.x hosts. This issue is often accompanied by heartbeat timeout errors in the system logs.

Symptoms - 

• Transferring large files (e.g., 80GB+) takes significantly longer than expected (e.g., 9 hours vs 1 hour).
• Virtual machines residing on the local datastore experience latency or become unresponsive.
• In /var/log/vobd.log, the following entries appear: [vob.vmfs.heartbeat.timedout] ####-####-#### SPHBGPRVRQO3002_DS02 [esx.problem.vmfs.heartbeat.timedout] ####-####-#### SPHBGPRVRQO3002_DS02
• In /var/log/vmkwarning.log, latency warnings are observed: WARNING: HPP: HppScsiThrottleLogForDevice:585: Cmd 0x2a (...) to dev "naa.####" on path "vmhba1:C3:T111:L0" Failed
• ESXTOP shows high DAVG values (exceeding 1000ms) for the local storage device.

Environment

VMware ESXi 8.0.x / 9.x

Cause

The underlying storage disks are responding slowly to heartbeat I/O requests. This can be caused by:

1. Outdated controller firmware or a driver/firmware mismatch.
2. Physical hardware failure of the disks, cables, or RAID controller.
3. Excessive I/O load on the local datastore.

Resolution

1. Verify that the controller driver and firmware are at the recommended levels using the Broadcom Compatibility Guide
2. Perform a hardware health check using the DCUI or Open Manage tools to ensure no physical disk failures are reported.
3. Validate the integrity of the SAS/SATA cables connecting the motherboard to the RAID controller.
4. If the issue persists on a specific host while others work correctly with identical configurations, contact the hardware vendor to investigate potential motherboard or backplane defects.

Additional Information

For more details on heartbeat timeouts, see Understanding lost access to volume

To speak with a customer representative or a Support Engineer see Contact Support Scroll to the bottom of the page and click on your respective region.