Virtual Machines Registered on Multiple Hosts after Network Flapping in vSphere HA

Products

• VMware vSphere ESXi

Issue/Introduction

• Virtual Machines (VMs) appear registered on different ESXi hosts simultaneously ("split brain").
  
  
• vCenter Server reports VMs flapping between hosts or showing as "Disconnected/Unknown."
  
  
• "Ghost" VMs remain in the inventory of an isolated host even after HA has restarted them elsewhere.

Environment

• VMware ESXi 8.0.x
• VMware vCenter Server 8.0.x

Cause

Because the host cannot communicate its state, due to network isolation, vSphere HA triggers a failover.  HA registers and powers on the VM on a healthy host and a stale registration remains on the original host. This results in dual registration of the VM.

Resolution

Follow these steps to identify the lock owner and remove the stale VM process:

1. Log in to the ESXi host CLI via SSH.
2. Identify which host holds the file lock by running the following command on the .vmx.lck file:

   vmkfstools -D /vmfs/volumes/####/VM_NAME/VM_NAME.vmx.lck

3. Locate the MAC address of the lock owner in the "owner" section of the output (i.e. the final 12 digits of the owner uuid, ####-####-####-####-############).
4. On the host that does not own the lock, identify the World ID of the "ghost" VM:

   esxcli vm process list

5. Forcefully kill the stale VM process:

   esxcli vm process kill -t=force -w=WorldID

6. Unregister the stale VM from the host:

   vim-cmd vmsvc/unregister VMID

7. To prevent recurrence, navigate to Cluster > Configure > vSphere Availability and ensure Host Isolation Response is set to Power off and restart VMs.