Putting ESXi into Maintenance mode, vMotioning or reconfiguring a VM fails due to the vmwarelwd ioFilter being attached

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:
Errors manifest in vmware.log when attempting to power on a VM, the VM is being vMotioned to another host, or a Fast-Suspend and Resume (FSR)
operation is performed. FSR is used when live-migrating VM storage with Storage vMotion, but also for VM Hot Add. Hot Add is the capability to add vCPUs, memory, and other selected VM hardware devices to a powered-on VM.

Errors visible in the vmware.log file:

• Module Disk power on failed. Cannot open the disk '/vmfs/volumes/datastore1/vm-1/vm-1.vmdk' or one of the snapshot disks it depends on. Operation failed.
• The migration has exceeded the maximum switchover time of 100 second(s)LWD: Failed to connect IPC client to serve request 23409792 for disk 0; error: Connection timed out in vmware.log

Environment

VMware vSphere ESXi 7.0.3
VMware vSphere ESXi 8.0.x

Cause

This issue may arise when a protected VM is migrated to a host where the DPD service is not running or where lightweight data protection is not functioning.

The ioFilter (named vmwarelwd), that is leveraged by VMware Cloud Disaster Recovery and other data protection solutions, might also still be attached to a VM when the solution is no longer used to protect it.

Resolution

Option 1
In clusters where DRS is enabled, its important to ensure that the DPD service on each host the LWD-protected VMs can migrate to is running and functional. If there are hosts within the cluster that are not enabled for lightweight data protection or can not support vmwarelwd for some reason, use anti-affinity rules to block LWD-protected VMs from migrating to them. This applies to manual migrations as well.

You may also need to ensure the DPD service is configured to start with the host from the vSphere Client.

1. Select the ESXi host in the inventory.
2. Click the Configure tab.
3. Under System, click Services.
4. Select the dpd service and use the Edit Startup Policy button to change it to "Start and Stop with host"

Option 2
For situations where it is no longer desired for the virtual machine to be protected by vmwarelwd-related solutions, you can use the following instructions to remove the IO filter from them. A full-sync will be necessary should protection be re-enabled on the VM at a later point.

The attached script (lwd-cleanup-util.py) is a pyVmomi-based utility that allows users to:

1. List LWD-protected virtual machines in vCenter inventory list
2. Disable LWD protection for selected virtual machines on a vSphere vCenter server where the problematic VM resides.

The script requires the VMware vSphere API Python Bindings. In particular, it requires:

• pyVim (vm, folder, connect)
• pyVmomi (with vim version of at least vim.version.v7_0_2_1)
• six (a single-file opensource dependency used by pyVim and pyVmomi)

All of these dependencies are satisfied by the VMware vCenter Appliance. Move the script into the vCenter Appliance, SSH to it, and execute it using the directions below.
 

Execution Steps

Help

Command Line Help

python lwd-cleanup-util.py --help

Usage

List all LWD-protected VM(s) which are protected by any vendors (including VMs protected by a vendor without a vendor ID).

python ./lwd-cleanup-util.py -u '[email protected]' --list
python ./lwd-cleanup-util.py -u '[email protected]' --list --all-vendors

List all LWD-protected VM(s) which are protected by 'myVendor'.

python ./lwd-cleanup-util.py -u '[email protected]' --list -i myVendor

List all LWD-protected VM(s) which are protected but do not have an associated vendor ID.

python ./lwd-cleanup-util.py -u '[email protected]' --list -i ""

Disable LWD protection provided by 'myVendor' if present on the specified VM.

python ./lwd-cleanup-util.py -u '[email protected]' -d -i myVendor -m test-vm-001

Disable LWD protection provided by a vendor which didn't specify a vendor ID, if present on the specified VM.

python ./lwd-cleanup-util.py -u '[email protected]' -d -i "" -m test-vm-001

Disable LWD protection provided by any vendor, if present on the specified VM.

python ./lwd-cleanup-util.py -u '[email protected]' -d --all-vendors -m test-vm-001

Disable LWD protection provided by 'myVendor' if present on the specified VMs.

python ./lwd-cleanup-util.py -u '[email protected]' -d -i myVendor -m vm1 -m vm2 -m vm3

Disable LWD protection provided by 'myVendor' for all VMs.

python ./lwd-cleanup-util.py -u '[email protected]' -d -i 'myVendor' --all

Connect to a remote VC '10.1.2.34' to list LWD protected VMs.

python ./lwd-cleanup-util.py -r 10.1.2.34 -u '[email protected]' --skipCertCheck -l

Connect to a remote VC '10.1.2.34' to disable LWD protection provided by 'myVendor' for all LWD-protected VMs.

python ./lwd-cleanup-util.py -r 10.1.2.34 -u '[email protected]' -d -i myVendor -all

Use custom pyVpx library from 'pyVpx' subfolder to connect to a remote VC and disable LWD protection provided by 'myVendor' for all LWD-protected VMs.

python ./lwd-cleanup-util.py -r 10.1.2.3 -u '[email protected]' -d -i myVendor --all --pyVpx pyVpx

Additional Information

Impact/Risks:
Removal of the vmwarelwd filter from a VM that is being protected will result in a full-sync should protection be re-enabled on the VM at a later point. The execution of the attached script should be performed only against VMs that are no longer to be protected by VCDR or other solutions that leverage the vmwarelwd filter.
The attached utility will not reconfigure a VM to remove any filter other than that of the vmwarelwd ioFilter.