Troubleshooting ESXi Host Remediation Tasks Hung or Failing Maintenance Mode

Products

• VMware vSphere ESXi

Issue/Introduction

• Remediate Host tasks in vSphere Lifecycle Manager (vLCM) or Update Manager (VUM) are stuck at 22% or fail.
• The ESXi host is not part of a Distributed Resource Scheduler (DRS) enabled cluster, or is a standalone host.
• There are similar errors seen in the log /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server-##.log:
  

  20##-##-##T##:##:##.###-##:## info vmware-vum-server[#####] [Originator@#### sub=Telemetry] [TelemetryManager ###] Sending telemetry data: {"@type":"pman_error_report","taskId":"########-####-####-####-############|########-####-####-####-############","entityId":"########-####-####-####-############|host-######","parentTaskId":"","errorMessageId":"com.vmware.vcIntegrity.lifecycle.health.vc.query_what_if_enter_mm.query_what_if_enter_mm_preCheckFailure","errorMessage":"DRS reported issues that prevent host '<HOST_FQDN_OR_IP>' from entering maintenance mode.","errorTime":"20##-##-##T##:##:##.######Z"}

  20##-##-##T##:##:##.###-##:## error vmware-vum-server[#####] [Originator@#### sub=EHP opID=########-####-####-####-############] [com.vmware.vcIntegrity.lifecycle.health.vc.query_what_if_enter_mm]: Host '<HOST_FQDN_OR_IP>' reported an issue which prevents entering maintenance mode: The operation is not allowed in the current state.

  20##-##-##T##:##:##.###-##:## info vmware-vum-server[#####] [Originator@#### sub=Telemetry] [TelemetryManager ###] Sending telemetry data: {"@type":"pman_error_report","taskId":"########-####-####-####-############|########-####-####-####-############","entityId":"########-####-####-####-############|host-######","parentTaskId":"","errorMessageId":"com.vmware.vcIntegrity.lifecycle.health.vc.query_what_if_enter_mm.query_what_if_enter_mm_host_fault","errorMessage":"Host '<HOST_FQDN_OR_IP>' reported an issue which prevents entering maintenance mode:  Cannot put host into maintenance mode since host is not in cluster and has # powered on virtual machines.","errorTime":"20##-##-##T##:##:##.######Z"}

Cause

Standalone ESXi hosts, or ESXI hosts that are not in a Distributed Resource Scheduler (DRS) enabled cluster, do not have access to Distributed Resource Scheduler (DRS) to automatically migrate virtual machines to other hosts. Since remediation tasks require the host to enter Maintenance Mode, the process hangs or fails because the running VMs have no automated evacuation path.

Resolution

To successfully remediate a standalone host, manual VM management is required to allow the host to enter Maintenance Mode:

1. Hosts in a Non Distributed Resource Scheduler (DRS) cluster:
   • Manually vmotion the virtual machines to other hosts in the cluster.
   • Proceed with steps 3-6.
2. For Standalone ESXi Host:
   • Schedule a Maintenance Window as a VM downtime window is necessary as VMs must be powered off or suspended.
   • Manually power off or suspend all running virtual machines on the host.
   • Proceed with steps 3-6.
3. Once all VMs are stopped or moved, right-click the host and select Maintenance Mode > Enter Maintenance Mode.
4.  Start the Remediate task in Lifecycle Manager. The task should now proceed with the host in Maintenance Mode.
5. After the remediation and any required reboots complete, right-click the host and select Maintenance Mode > Exit Maintenance Mode.
6. Manually power on, resume, or migrate back the virtual machines.

Additional Information

If using NSX see KB: Upgrading NSX on Standalone Hosts with unmovable workload.

If using Zerto see KBL Unable to remediate the host at cluster level due to "unable to put the host in maintenance mode" error 

If the remediation task remain stuck in the vCenter UI even after powering off of moving the VMs, it may be necessary to restart the Update Manager service: `service-control --restart vmware-updatemgr`