Entering Maintenance Mode task stuck or does not progress and fails for VMFS, NFS and vVols datastores

Products

• VMware vSphere ESXi

Issue/Introduction

This article provides required steps to successfully complete datastore maintenance workflows which might be impacted due to the placement of vCLS VMs.

Symptoms:
Datastore enter-maintenance mode tasks might be stuck for long duration as there might be powered on vCLS VMs residing on these datastores. vCLS VM placement is taken care of by the vCenter Server, so user is not provided an option to select the target datastore where vCLS VM should be placed. This datastore selection logic for vCLS VMs selects a shared datastore for a cluster if there is one available/connected with enough capacity to place these VMs. 

The datastore selected for Maintenance Mode might be deployed with vSphere Cluster Services VMs which cannot be powered off. The presence of these VMs on this datastore can prevent it from entering maintenance mode.

For more information, see vSphere Cluster Services (vCLS) in vSphere 7.0 Update 1 (80472).

Environment

VMware vSphere ESXi 7.0.0

Cause

This issue occurs as with the release of vSphere Cluster Services features in vSphere 7.0 Update 1, there are a set of datastore maintenance workflows that could require some manual steps by users, as vCLS VMs might be placed in these datastores which cannot be automatically migrated or powered off.  

Note: vCLS VMs are not supported for Storage DRS.

Resolution

To resolve this issue:

Prior to placing a datastore in Maintenance Mode, check if there are any vCLS VMs deployed in that datastore. If there are, migrate those VMs to another datastore within the cluster if there is another datastore attached to the hosts within the cluster. 

Alternatively,

If the datastore that is being considered for maintenance is the only datastore connected to the host and this datastore has vCLS VM, then the only option is to power-off this vCLS VM is by using Retreat Mode on the cluster. 

Note: This datastore might be connected to different hosts in different clusters and vCLS VMs from all these different clusters might be stored in this datastore. In such a case, retreat mode has to configured on all the clusters connected to this datastore, so this datastore can be taken down for maintenance. 

• Use Retreat Mode to remove vSphere Cluster Service VMs from a cluster. First enable Retreat Mode in the cluster so that all the vCLS VMs in the cluster gets deleted. If in case the same datastore is shared between multiple clusters and the vCLS VMS from different clusters are placed in this same datastore, then in all the clusters where the corresponding cluster vCLS VMs are placed in this datastore should be enabled with Retreat Mode. Also when this Retreat Mode is enabled the DRS in that particular clusters will be non-functional until the Retreat Mode is disabled back. Once the datastore Maintenance Mode is successful the retreat mode should be disabled in all clusters where its enabled in order to get back the vCLS VMs in all the cluster for DRS functionality.