A VM crashes and becomes inaccessible due to loss of underlying VM files

Products

• VMware vSphere ESXi

Issue/Introduction

VM/s crash and go into an inaccessible/orphaned state.

A check on the contents of the VM folder indicates that vmx, vmdk descriptors and vmware.log files are missing:

[root@<Hostname>:/vmfs/volumes/<Datastore-Name>/<VM-Name>] ls -lah
total 10949572
drwxr-xr-x    2 root     root        4.0K May XX 10:05 .
drwxr-xr-x  210 111037   25         20.0K May XX 09:14 ..
-rw-------    1 root     root      300.0G May XX 02:45 VM-NAME_1-flat.vmdk
-rw-------    1 root     root        6.8M May XX 06:54 vmmcores.gz
-r--------    1 root     root       10.5M May XX 06:54 vmx-zdump.000

In the working directory of a vm, its expected we would see key vm files such as:
e.g. vmx, vswps, disk descriptors (<VM-Name>.vmdk), base disks (<VM-NAME>-flat.vmdk) and vmware.logs

Environment

VMware vSphere ESXi 7.0
VMware vSphere ESXi 8.0

 

Cause

 

When the ESXi host attempts to access the vm files and it is unable to do so, logging similar to below will be seen in the hostd.log.

ESXi: /var/log/hostd.log

####-##-##T##:##:##.###Z In(166) Hostd[2099237]: [Originator@6876 sub=Libs] DictionaryLoad: Cannot open file "/vmfs/volumes/<Datastore-Name>/<VM-Name>/<VM-Name>.vmx": No such file or directory.

####-##-##T##:##:##.###Z Db(167) Hostd[2099241]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/<Datastore-Name>/<VM-Name>/<VM-Name>.vmx] Handling vmx message17308716: The lock protecting '<VM-Name>.vmdk' has been lost, possibly due to underlying storage issues.

The VM can crash if vm is running and encounters underlying storage issues that cause vm descriptors or other files to be lost. 

This can also occur if non locked vms files are inadvertently while the vm is running (ie via the datastore browser).

 

Resolution

If only descriptor files are missing they can be recreated using the KB below:

Recreating a missing VMware virtual machine disk descriptor file (.vmdk)

If files have been inadvertently moved using the datastore browser they may be moved back to the original directories.

If vmdk flat files are missing then contact Broadcom support.

Note: It may be necessary to restore the relevant vm/files from backup.