'Enter maintenance mode' task stuck at 22% in Supervisor enabled cluster's ESXi host

Products

• VMware vSphere Kubernetes Service
• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• Attempting to place an ESXi host into maintenance mode hangs at 22%.
• The vCenter task status shows 'Reregistering powered off VMs on another host'.
• The vSphere Client "VMs" tab for the host shows no active virtual machines.
• /var/log/vmware/vpxd/vpxd.log on vCenter contains entries indicating the host still has PodVMs:

YYYY-MM-DDThh:mm-ss info vpxd[####] [Originator@#### sub=MoHost opID=####-####-####-####] <esxi-host-fqdn> has PodVM [vim.VirtualMachine:vm-##,<PodVM-name>]
YYYY-MM-DDThh:mm-ss info vpxd[####] [Originator@#### sub=MoHost opID=####-####-####-####] <esxi-host-fqdn> has PodVM [vim.VirtualMachine:vm-##,<PodVM-name>]

Environment

• vCenter Server 8.x
• vSphere with Tanzu / Supervisor Cluster

Cause

The Workload Control Plane (WCP) failed to cleanly decommission or migrate vSphere Pods from the host. Because these PodVMs remain in the ESXi host inventory, vCenter attempts to re-register them to another host as part of the maintenance mode workflow, but the process stalls if the Pods are in an invalid or orphaned state.

Resolution

1. Log in to the affected ESXi host via SSH and run the below command:
   
   vim-cmd vmsvc/getallvms
   
   Locate the VMid for the PodVMs reported in the /var/log/vmware/vpxd/vpxd.log.
   When checking these pods in the Supervisor via "kubectl get pods", they are in Pending or NotFound state.
   
   
2. Manually remove the stale Pod entries from the host inventory after ensuring it is no longer needed:
   
   vim-cmd vmsvc/unregister <VMID>
   (Note: Replace <VMID> with the actual ID found in step 1.)
   
   
3. Check the vCenter inventory for any orphaned vSphere Pods by checking for "(orphaned)" in their name.
   If so, determine if those are needed. If not, unregister them via vCenter.
4. The vCenter task may remain in a "Running" state even after unregistering the VMs. To clear the task, restart the vpxd service on the vCenter Server Appliance by running the below command:
   
   service-control --restart vmware-vpxd
   
   
5. Once vpxd is back online, retry the "Enter Maintenance Mode" task.