Fluctuating Virtual Machine Count and Maintenance Mode Failure on ESXi Host

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

•  Virtual machine count fluctuates continuously when viewed via the vSphere Client or ESXi Host Client (Summary or Virtual Machine menus).

• The affected ESXi host persistently fails to complete the Enter Maintenance Mode task.

• Virtual machine migration attempts fail and return invalid state errors.

Environment

VMware vCenter server 7.0
VMware vCenter server 8.0
VMware vSphere ESXi 7.0
VMware vSphere ESXi 8.0

Cause

Same virtual machine is running on two different ESXi hosts due to abnormal HA behavior before the problem happens

Resolution

1. Select one unstable virtual machine as example.
2. Run command "localcli vm process list|grep <VM name>" on all ESXi hosts in the cluster
3. Take a note of the world ID of the non-responsive virtual machine. 
4. Power off the virtual machine from the list by running one of these commands:
   
   esxcli vm process kill -t=soft -w=WorldID
   
     or
   
   esxcli vm process kill -t=hard -w=WorldID
   
     or
   
   esxcli vm process kill -t=force -w=WorldID
5. If the VM fails to stop even after running the "esxcli vm process kill" command, evacuate all other VMs off the ESXi host and restart the ESXi itself.

Reference: Powering off an unresponsive virtual machine on an ESXi host 

Additional Information

Impact/Risks:
This symptom has no impact to service