Host Maintenance Mode Operation task stuck or does not progress

Products

• VMware vSphere ESXi

Issue/Introduction

• Host Maintenance Mode Operation task stuck at Namespaces Maintenance Mode step.
• In the /var/log/vmware/wcpsvc.log, you see messages similar to:

2020-11-10T09:57:20.631Z error wcp [opID=cluster-xxx-host-yyy] Failed to uninstall vibs on host HostSystem:host-yyy. Err InstallationError: The installation transaction failed. Code=15 Message=Failed to query file system stats: Errors: Cannot open volume: /vmfs/volumes/vsan:xxxxx
OR
2020-11-10T20:42:27.901Z error wcp [opID=cluster-xxx-host-yyy] Failed to set Spherelet config: Cannot change the host configuration.

Environment

VMware vSphere ESXi 7.0.x

Resolution

Host Maintenance Mode Operation could be blocked infinitely if this issue occurs. Normally, PatchManager expects that the live environment is consistent with the bootbank. If vib removal passed live removal phase (tardisk is gone), but failed at bootbank change phase (which usually is the lower risk), it can lead to inconsistency. Then, if after the initial failure no reboot has been done, PatchManager can report wrong information about the state of the vib. To workaround the issue, reboot the host involved.