ESXi Maintenance Mode Stalls in VKS Cluster Due to Stale PodVMs

Products

• VMware

Issue/Introduction

• The affected ESXi host is in a cluster where Supervisor/VKS is configured.
• The VM tab in the vCenter UI for the ESXi host shows that no VMs are running on it.
• The 'enter maintenance mode' task shows 'Waiting for all VMs to be powered off or suspended or migrated' in the details section.
• In an SSH session to the ESXi host, the following command shows that there is an envoy-#### PodVM running on the host.
  vim-cmd vmsvc/getallvms
vmid      Name
##        envoy-####


• In the ESXi host UI, the above envoy-#### PodVM shows as powered off
• In /var/log/vmware/vpxd/vpxd.log the host already seems to be in maintenance mode per error below.
  Error:
-->    com.vmware.vapi.std.errors.already_in_desired_state
--> Messages:
-->    vcenter.wcp.node.alreadyindesiredstate<Node identified by host-## is already in state NodeMaintenance

Environment

• vCenter Server 8.x
• Supervisor
• vSphere Kubernetes Service

Cause

• Workload Control Plane (WCP) fails to decommission the PodVM automatically when the host enters maintenance mode, hence the task gets stuck.

Resolution

The stale PodVM needs to be manually deleted in order for the host to enter maintenance mode. 

Note: Perform a backup or take an offline snapshot of the vCenter Server Appliance before proceeding. If the vCenter Server is in Enhanced Linked mode, backups or offline snapshot has to be created for every vCenter server as described in KB Snapshot Best practices for vCenter Server Virtual Machines

1. Open an SSH session to the affected ESXi host and login as the root user.
2. Run the below command to get the vmid of the envoy PodVM.
   vim-cmd vmsvc/getallvms

Note: Make a note of the vmid of the affected PodVM.
3. Run the below command to remove the terminating PodVM using the vmid.
   vim-cmd vmsvc/destroy <vmid>


4. Check the vCenter UI if the VM still exists in it.
5. If yes, the PodVM has to be removed from the vCenter database.
6. Open an SSH session to the vCenter server and login as the root user.
7. Stop the vmware-vpxd and wcp services.
   service-control --stop vpxd
service-control --stop wcp
   
   
8. Login to the vCenter server database with the below command:
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB
   
   
9. Query the ID for the affected VM (replace <VM-Name> to the actual VM name) in the VPX_ENTITY table:
   select * from vpx_entity where name like '%<VM-Name>%';
   
   
10. Delete the VM_ID from the following tables in the same order:
    delete from VPX_COMPUTE_RESOURCE_DAS_VM where VM_ID=####;
delete from VPX_COMPUTE_RESOURCE_DRS_VM where VM_ID=####;
delete from VPX_COMPUTE_RESOURCE_ORC_VM where VM_ID=####;
delete from VPX_VM_SGXINFO where VM_ID=####;
delete from VPX_GUEST_DISK where VM_ID=####;
delete from VPX_VM_VIRTUAL_DEVICE where ID=####;
delete from VPX_VM_DS_SPACE where VM_ID=####;
delete from VPX_NON_ORM_VM_CONFIG_INFO where ID=####;
delete from VPX_NORM_VM_FLE_FILE_INFO where VM_ID=####;
delete from VPX_VDEVICE_BACKING_REL where VM_ID=####;
delete from VPX_VIRTUAL_DISK_IOFILTERS where VM_ID=####;
delete from VPX_VM_STATIC_OVERHEAD_MAP where VM_ID=####;
delete from VPX_VM_TEXT where VM_ID=####;
delete from VPX_VM where ID=####;
delete from VPX_ENTITY where ID=####;

Reference KB: Manually removing a stale VM from the vCenter Server vpostgres database
    
    
11. Exit the vCenter database with the below command:
    \q
    
    
12. Start the stopped vCenter services with the below commands:
    service-control --start vpxd
service-control --start wcp
    
    
13. Login to the vCenter Server UI, the affected envoy PodVM will not exist.
14. Place the host in maintenance mode.