Error: Input/output error when registering or powering on VMs in ESXi

Products

• VMware vSphere ESXi

Issue/Introduction

• Virtual machines (VMs) appear in an Invalid or Inaccessible state in the VMware ESXi host client or vCenter Server.
• There has been a critical hardware failure requiring a motherboard replacement, or an ungraceful host shutdown (PSOD).
• Attempts to register the VM or power it on fail with file system errors.
  • Error: FILE:open error on ... .vmx: Input/output error.
• Files on the datastore are visible in a directory listing but cannot be copied or modified via WinSCP or CLI.

Environment

VMware vSphere ESXi (All versions)

Cause

When a host fails abruptly, or when system changes occur (e.g., a motherboard replacement), the VMFS datastores may retain inconsistent metadata or stale 'heartbeat' locks. This appears as metadata corruption or locked resources, preventing the ESXi host from correctly accessing the volume.

In the /var/log/VMkernel file, you see entries similar to:

vmkernel: 25:21:39:57.861 cpu15:1047)FS3: 130: <START vm.vswp>
vmkernel: 25:21:39:57.861 cpu15:1047)Lock [type 10c00001 offset 52076544 v 69, hb offset 4017152
vmkernel: gen 109, mode 1, owner xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx mtime 3420]
vmkernel: 25:21:39:57.861 cpu15:1047)Addr <4, 1011, 10>, gen 36, links 1, type reg, flags 0x0, uid 0, gid 0, mode 600
vmkernel: 25:21:39:57.861 cpu15:1047)len 3221225472, nb 3072 tbz 0, zla 3, bs 1048576
vmkernel: 25:21:39:57.861 cpu15:1047)FS3: 132: <END vm.vswp>
vmkernel: 0:00:20:51.964 cpu3:1085)WARNING: Swap: vm 1086: 2268: Failed to open swap file '/volumes/xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx/vm/vm.vswp': Invalid metadata
vmkernel: 0:00:20:51.964 cpu3:1085)WARNING: Swap: vm 1086: 3586: Failed to initialize swap file '/volumes/xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx/vm/vm.vswp': Invalid metadata
cpu11:268057)WARNING: HBX: 599: Volume xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx ("<Datastore name>") may be damaged on disk. Corrupt heartbeat detected at offset 3305472: [HB state 0 offset 6052837899185946624 gen 15439450 stampUS 5 $

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

 

Resolution

Engage Broadcom support to attempt to repair the detected corruption, using vSphere On-disk Metadata Analyzer (VOMA) 

Warning: Restore from backup may be required, where corruption is not repairable.

Additional Information

Related links:

Using vSphere On-disk Metadata Analyzer (VOMA) to check VMFS metadata consistency

Error: Virtual machine inaccessible or invalid due to ESXi host file lock

Virtual machines appear as invalid or orphaned or inaccessible