I/O access on Virtual Machine File System returns invalid metadata error

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Each VMware vSphere VMFS host maintains information in the form of an on-disk heartbeat. After reading a heartbeat slot, the heartbeat was found to contain invalid data.

Example

A VMFS datastore, datastore-1, resides on a SAN LUN. HostA connects to the SAN through a Fibre Channel. HostB also connects to the SAN through the Fibre Channel. Several VMs are running on HostA , and contain disk files residing on datastore-1. On HostB, a user inadvertently issues a dd command directly on the LUN, zeroing out all VMFS metadata. The next time HostA reads its own heartbeat slot, it sees all zeroes, determines that the heartbeat is corrupt, and this event occurs.

Impact

This host can no longer establish a heartbeat to the volume. No further metadata operations can be performed. Any attempt to open the volume fails if the host reads a corrupt heartbeat slot while acquiring a heartbeat.

Environment

VMware vSphere ESXi 6.5
VMware vCenter Server 6.5.x

Resolution

Repair invalid heartbeat data with the vSphere On-disk Metadata Analyzer (VOMA) utility. See Using vSphere On-disk Metadata Analyzer (VOMA) to check VMFS metadata consistency (2036767) for more information.