Error "Operation failed Failed to start the virtual machine. Cannot open the disk" while powering on a virtual machine

Products

VMware vSphere ESXi VMware Live Recovery

Issue/Introduction

Symptoms:

Powering on the VM fails with the below error in the vCenter UI:

Operation failed Failed to start the virtual machine. Cannot open the disk

You may also get the below error while powering in the virtual machine in the vCenter UI:

Power on Error: Cannot open the disk '/vmfs/volumes/datastore_name/vm_directory/xxxxxxxxxxxxxx.vmdk' or one of the snapshot disks it depends on.
Module 'Disk' power on failed.

vMotion, Migration and Cloning of the VM or the disks will fail as well at the stage where the vmdk is being opened due to the same error
The virtual machine is currently or has in the past been protected by a solution that uses the LWD filter. (VCDR, Dell PPDM, etc..)

Steps to Validate:

Messages similar to the following are seen in the virtual machines vmware.log file located in '/vmfs/volumes/datastore_name/vm_directory/'.

YYYY-MM-DDTHH:MM:SSZ Er(02) worker-zzzzzzzz LWD: Hash mismatch for bitmap (sidecar version 3)
YYYY-MM-DDTHH:MM:SSZ Er(02)+ worker-zzzzzzzz expected: 89ac7da06e81c89cb982380d4a90c6d17f239a82
YYYY-MM-DDTHH:MM:SSZ Er(02)+ worker-zzzzzzzz computed: 5e15a609e14d3b964aba2ac5d5c7525ef2dfdb70
YYYY-MM-DDTHH:MM:SSZ Wa(03) worker-zzzzzzzz LWD: Hash mismatch while upgrading bitmap hash for type 1, offset 688128 in sidecar YYYYYYYYY
YYYY-MM-DDTHH:MM:SSZ Er(02) worker-zzzzzzzz LWD: Could not read the disk details for disk YYYYYYYYY from the sidecar 1: Operation failed
YYYY-MM-DDTHH:MM:SSZ In(05) worker-zzzzzzzz LWD: LwdFilter_Exit while on disk YYYYYYYYY
YYYY-MM-DDTHH:MM:SSZ In(05) worker-zzzzzzzz IOFIPC: IPC Service is no longer accepting connections on 86
YYYY-MM-DDTHH:MM:SSZ In(05) worker-zzzzzzzz IOFIPC: IPC management subsystem shut down

Disk Consistency check will show that the disk chain is consistent. However, it will also mention the below:

[#]vmkfstools -e /vmfs/volumes/datastore_name/vm_directory/xxxxxxxxxxxxxx.vmdk

LWD: Disk YYYYYYYYY was not closed cleanly, a full sync will be required
LWD: Disk YYYYYYYYY requires full sync
Disk chain is consistent.

Environment

VMware vSphere ESXi 8.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 6.x

VM has its disks protected by solutions such as VMware Live Cyber Recovery (vLCR) or Dell PPDM or other replication solutions

Cause

This issue occurs when there is a hash (string used to compare data) mismatch in the bitmap, which is caused by corruption or inconsistency in the IO Filter Metadata used for VMware's Lightweight Delta(LWD) protection. The below log line in /vmfs/volumes/datastore_name/vm_name/vmware.log confirms the same:

YYYY-MM-DDTHH:MM:SSZ Er(02) worker-zzzzzzzz LWD: Hash mismatch for bitmap (sidecar version 3)

The issue can be caused after unexpected shutdowns, storage issues or due to metadata synchronization failures between the protected VM and the replication mechanism

Resolution

SSH to the Host where the VM resides
Get details of the VM using the following command:

[#]vim-cmd vmsvc/getallvms | grep "vm name"

Navigate to the directory of the VM using the path obtained in above step:

[#]cd /vmfs/volumes/datastore_name/vm_directory

Create a temporary directory inside the directory for the VM, e.g.

[#/vmfs/volumes/datastore_name/vm_directory] mkdir removeme

Move the vmfd file for the Disk's for the VM to directory created above, e.g.

[#/vmfs/volumes/datastore_name/vm_directory]mv *.vmfd removeme

Note: The vmfd file is contains metadata for VMDKs and is created when the VM has its disks protected by solutions such as VMware Cloud Disaster Recovery (VCDR) or Dell PPDM or other replication solutions

Remove LWD protection for the VM using the PowerCli script mentioned in the KB Putting ESXi into Maintenance mode, vMotioning or reconfiguring a VM fails due to the vmwarelwd ioFilter being attached
Power on the VM
Enable protection again using LWD and ensure a full backup or a full synchronization is performed
Once the VM is up and protection is re-enabled, delete the temporary folder created and its contents

Additional Information

Remove the ctk and the vmsd files too by moving them to the temporary directory created above if VM is failing to power on with error Invalid Change Tracking File or with error Could not open/create change tracking files:

Could not open/create change tracking files