Virtual machine reports as invalid with missing objects and directory on vSAN datastore
search cancel

Virtual machine reports as invalid with missing objects and directory on vSAN datastore

book

Article ID: 422864

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

Symptoms:

  • A virtual machine is displayed as invalid in the vCenter server and Host Client.

  • Searching for vSAN objects by the virtual machine's display name returns no results.

  • The VM directory appears to be missing from the vSAN datastore

  • While the initial symptoms suggests the virtual machine files are deleted, further investigation reveals the virtual machine's display name and its actual directory name on the vSAN datastore is different.

    This can be verified using the below command:

    esxcli vm process list
    VM1:
       World ID: 2119550
       Process ID: 0
       VMX Cartel ID: 2119546
       UUID: 42 29 67 54 50 01 98 59-7d a1 bc 3d 25 72 5f e6
       Display Name: VM1
       Config File: /vmfs/volumes/vsan:52xxxxxxxxxxxxxx-9116d31e31e1bbc8xxxxxxxxxxxxxxxx/897f5466-d888-f22e-b1ba-xxxxxxxxxxxx/VM2.vmx

Environment

VMware VSAN 8.x

Cause

This issue is caused by a vSAN Cluster partition. When an ESXi host becomes isolated from the rest of the cluster, it loses access to the distributed storage quorum.

Even after network connectivity is restored, the host may retain a stale Lock on the VM configuration or swap files. Because the files are locked by the isolated host's management agent, the vCenter Server cannot properly register or initialize the VM, leading to it being flagged as Invalid.

Cause Validation:

Analysis of /avr/run/log/hostd.log confirms that the virtual machine was marked as invalid immediately following a storage inaccessibility event on the backend.

2025-12-08T16:11:31.719Z Er(163) Hostd[2355666]: [Originator@6876 sub=Vmsvc.VMFileChecker] Unable to stat VM config file '/vmfs/volumes/vsan:524cf99ba9e094e3-xxxxxxxxxxxx/897f5466-xxxx-xxxx-xxxx-xxxxxxxxxxxx/VM2.vmx'. Error code 5: 'Input/output error'
2025-12-08T16:11:31.719Z Wa(164) Hostd[2355666]: [Originator@6876 sub=Vmsvc] LoadVm file check error: Unable to stat VM config file '/vmfs/volumes/vsan:524cf99ba9e094e3-xxxxxxxxxxxx/897f5466-xxxx-xxxx-xxxx-xxxxxxxxxxxx/VM2.vmx'. Error code 5: 'Input/output error'
2025-12-08T16:16:13.359Z In(166) Hostd[2355666]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:524cf99ba9e094e3-xxxxxxxxxxxx/897f5466-xxxx-xxxx-xxxx-xxxxxxxxxxxx/VM2.vmx] UpdateStorageAccessibilityStatusInt: Vm's storage accessibility status changed to false

2025-12-08T16:16:24.379Z In(166) Hostd[2355684]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:524cf99ba9e094e3-xxxxxxxxxxxx/897f5466-xxxx-xxxx-xxxx-xxxxxxxxxxxx/VM2.vmx] UpdateStorageAccessibilityStatusInt: Vm's storage accessibility status changed to true

2025-12-08T16:17:54.440Z Wa(164) Hostd[2355684]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:524cf99ba9e094e3-xxxxxxxxxxxx/897f5466-xxxx-xxxx-xxxx-xxxxxxxxxxxx/VM2.vmx] Failed to load virtual machine: Fault cause: vim.fault.GenericVmConfigFault
2025-12-08T16:17:54.440Z Wa(164) Hostd[2355033]: -->
2025-12-08T16:17:54.440Z In(166) Hostd[2355684]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:524cf99ba9e094e3-xxxxxxxxxxxx/897f5466-xxxx-xxxx-xxxx-xxxxxxxxxxxx/VM2.vmx] Marking VirtualMachine invalid
2025-12-08T16:17:54.440Z In(166) Hostd[2355684]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:524cf99ba9e094e3-xxxxxxxxxxxx/897f5466-xxxx-xxxx-xxxx-xxxxxxxxxxxx/VM2.vmx] State Transition (VM_STATE_INITIALIZING -> VM_STATE_INVALID_CONFIG)

Analysis of /var/run/log/vsansystem.log reveals that the storage inaccessibility was caused by a network partition that reduced the host's cluster membership to a nodeCount of 1, resulting in an isolated state.

2025-12-08T16:01:45.418Z In(166) vsansystem[2100534]: [vSAN@6876 sub=VsanSystemProvider opId=refresh-ba43] Complete, nodeCount: 1, runtime info: (vim.vsan.host.VsanRuntimeInfo) 
2025-12-08T16:16:08.879Z In(166) vsansystem[2100534]: [vSAN@6876 sub=VsanSystemProvider opId=refresh-babb] Complete, nodeCount: 1, runtime info: (vim.vsan.host.VsanRuntimeInfo)

Following the resolution of the cluster partition, review of the /var/run/log/vmkernel.log confirmed that a stale lock persisted on the virtual machine's swap file (.vswp), preventing the VM from initializing correctly.

2025-12-08T16:50:47.206Z In(182) vmkernel: cpu4:2362821)FS3: 148: <START VM2.vswp.lck>
2025-12-08T16:50:47.206Z In(182) vmkernel: cpu4:2362821)Lock [type 10c00001 offset 140091392 v 15, hb offset 3571712
2025-12-08T16:50:47Z In(182) vmkernel: gen 3, mode 1, owner 69343224-xxxxxxxx-xxxx-b4xxxxxxxxxx mtime 32197
2025-12-08T16:50:47Z In(182) vmkernel: num 0 gblnum 0 gblgen 0 gblbrk 0]
2025-12-08T16:50:47Z In(182) vmkernel: <FD c60 r5> is on Head Extent
2025-12-08T16:50:47.206Z In(182) vmkernel: cpu4:2362821)Addr <4, 60, 5>, gen 10, links 1, type reg, flags 0x0, uid 0, gid 0, mode 600
2025-12-08T16:50:47Z In(182) vmkernel: len 0, nb 0 tbz 0, cow 0, newSinceEpoch 0 zla 4305, bs 65536, lfb 0, tbzGShift 20, parent <4, 0, 0>, affinity <4, 60, 5>, lastSFBC 0, las$
2025-12-08T16:50:47.206Z In(182) vmkernel: cpu4:2362821)FS3: 150: <END VM2.vswp.lck>

The highlighted MAC address in the logs corresponds to the physical network interface of the host currently holding the file lock.

 

Resolution

To resolve the invalid state, the file locks must be released to re-initialize the VM metadata.

Refer: Investigating Virtual Machine file locks on ESXi Host(s)

Powered by Wolken Software