Datastore fails to unmount via vCenter with Failed to communicate to the host / VMs fail to power off

Products

• VMware vSphere ESXi
• VMware Live Recovery

Issue/Introduction

• VMs fail to power off/shutdown
• Unmounting the datastore fails with the error: Failed to communicate with the host.
• The datastore is in read only mode
• The datastore is in the DR site of a Live Recovery pair
• In var/log/vmkernel.log from any of the hosts the datastore is mounted to you see the below messages:

0x7 0x27 means data protect / write protected

Environment

vSphere ESXi (All Versions)

VMware Live Site Recovery (All Versions)

Cause

This is caused due to the datastore is in read only mode and write protected due to not following the manual VM re-protect steps accurately for Array based replication. All VMs should be powered off/down before setting the datastore to read only. See KB No hosts with hardware version which are powered on and not in maintenance mode are available for complete/detailed steps

This means the vmx files can't be written to, to shutdown the VMs. This also means no VM functionality can be performed to get the VMs off of the datastore to be able to unmount it. As per Unmount Datastores no VMs should reside on the datastore prior to unmounting.

Resolution

Perform a cold rolling reboot of the cluster without placing the hosts into maintenance mode via the out of band host management system/KVM. This will force the VMs that reside on the read only datastore to be powered down.