Unable to access certain files on a VMFS datastore

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• You have problems accessing certain files on a VMFS datastore.
• You cannot modify or erase files on a VMFS datastore.
• Attempting to read files on a VMFS datastore may fail with the error:
  invalid argument
   
• You see entries in the /var/log/vmkernel file similar to:

vmkernel: 25:21:39:57.861 cpu15:1047)FS3: 130: <START termserv2-5160fe37.vswp>
vmkernel: 25:21:39:57.861 cpu15:1047)Lock [type 10c00001 offset 52076544 v 69, hb offset 4017152
vmkernel: gen 109, mode 1, owner 4a15b3a2-fd2f4020-3625-001a64353e5c mtime 3420]
vmkernel: 25:21:39:57.861 cpu15:1047)Addr <4, 1011, 10>, gen 36, links 1, type reg, flags 0x0, uid 0, gid 0, mode 600
vmkernel: 25:21:39:57.861 cpu15:1047)len 3221225472, nb 3072 tbz 0, zla 3, bs 1048576
vmkernel: 25:21:39:57.861 cpu15:1047)FS3: 132: <END termserv2-5160fe37.vswp>

vmkernel: 0:00:20:51.964 cpu3:1085)WARNING: Swap: vm 1086: 2268: Failed to open swap file '/volumes/4730e995-faa64138-6e6f-001a640a8998/mule/mule-560e1410.vswp': Invalid metadata
vmkernel: 0:00:20:51.964 cpu3:1085)WARNING: Swap: vm 1086: 3586: Failed to initialize swap file '/volumes/4730e995-faa64138-6e6f-001a640a8998/mule/mule-560e1410.vswp': Invalid metadata

cpu11:268057)WARNING: HBX: 599: Volume 50fd60a3-3aae1ae2-3347-0017a4770402 ("<Datastore_name>") may be damaged on disk. Corrupt heartbeat detected at offset 3305472: [HB state 0 offset 6052837899185946624 gen 15439450 stampUS 5 $

Environment

VMware vSphere ESXi 6.0
VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.5
VMware vSphere ESXi 7.0.0

Cause

VMFS is a journaling file system where it keeps track of changes not committed to the disk yet. These journals are later replayed by the hosts.

The journal/HB block leaks happen on VMFS Filesystem in the case of storage connectivity problems while closing volume.
 

Resolution

A VMFS datastore has a region designated for heartbeat types of operations to ensure that distributed access to the volume occurs safely. When files are being updated, the heartbeat region for those files is locked by the host until the update is complete. The invalid argument status indicates that the content of the heartbeat region is not correct. The subset of files managed by that region is likely to have been locked by a single host that crashed at some point (although other reasons are possible).

If a corrupt heartbeat region is at fault, an ESXi host that tries to access those files receives the same symptoms and the same HB offset value.

vmkfstools -D filename generates the same output.

The VMFS metadata is compromised and we cannot guarantee to repair it. Therefore we advise evacuating the device and reformatting a new datastore. For more information see vSphere On-disk Metadata Analyzer (VOMA) check VMFS metadata consistency and Checking Metadata Consistency with VOMA.

Additional Information

Note: In a rare scenario, one can have a situation where two hosts that share a common resource tries to replay a given journal in parallel. This can cause the journal not to be completely replayed and can cause certain locks (taken on the resource) to be lost. This in turn can cause a VM on the host to get hung or inaccessible.

For translated versions of this article, see:

• 日本語: 破損したハートビート領域： VMFS データストア上のファイルへのアクセスに関する問題 (1036184)

To be alerted when this document is updated, click the Subscribe to Article link in the Actions box.

破損したハートビート領域： VMFS データストア上のファイルへのアクセスに関する問題
Using vSphere On-disk Metadata Analyzer (VOMA) to check VMFS metadata consistency
Não é possível acessar determinados arquivos em um datastore VMFS
Imposible acceder a ciertos archivos de un almacén de datos VMFS