Deleted Virtual Machine objects getting "Discovered virtual-machine on esxi01.sampledomain.com. in Cluster01"

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

Virtual Machine’s that have had failed vSphere HA failovers in the past can retain their "vmFailoverInProgress" flag as 'true' in the Virtual Machine runtime. If a Virtual Machine is also registered to another ESXi host, vCenter will trigger the unregistration of the Virtual Machine on one of the ESXi hosts. However, vCenter will be unable to unregister the Virtual Machine due to 'vmFailoverInProgress=true', leaving the Virtual Machine registered on multiple hosts.

Now if the Virtual Machine is deleted or SvMotioned, it will not be possible to recreate the Virtual Machine with the same config path as it still has one stale copy registered on one of the ESXi hosts.

Environment

• VMware vSphere 7.0
• VMware vSphere 8.0
• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Cause

There may be two kinds of VMs in such a situation,

1. Virtual Machines' are already deleted or migrated by Storage vMotion - In such case, the other register instance will show as inaccessible at vCenter. We can unregister this instance from the ESXi host.
2. Some Virtual Machine's are still registered on two ESXi hosts and power-on. These Virtual Machine's can be identified by vpxd log 'Ignoring matched Vm that is currently undergoing failover'
   Sample log - 'Ignoring matched Vm that is currently undergoing failover. VM:[vim.VirtualMachine:vm-##,TestVm1] oldHost:[vim.HostSystem:host-###, 192.168.1.11] newHost:[vim.HostSystem:host-##, 192.168.1.12] and isPoweredOn:true'

Resolution

1. Unregister VMs that are inaccessible and known to have the vmFailoverInProgress flag
   • vim-cmd /vmsvc/getallvms
   • vim-cmd /vmsvc/get.runtime <vm-id> | grep -i "vmFailoverInProgress = true"
     Note: You can safely unregister the VMs that are showing as inaccessible and whose flag is true.
2. Check for ESXi hosts that have "ghost VMs" i.e. failed vSphere failoverszgrep -i "Ignoring matched Vm that is currently undergoing failover" /var/log/vmware/vpxd* | grep -i "isPoweredOn:false" | grep -E -o "newHost[^ ]+" | grep -E -o "nl[a-z0-9\.-]+" | sort | uniq
   • zgrep -i "Ignoring matched Vm that is currently undergoing failover" /var/log/vmware/vpxd* | grep -i "isPoweredOn:false" | grep -E -o "newHost[^ ]+" | grep -E -o "nl[a-z0-9\.-]+" | sort | uniq
3. This is a known issue that will be resolved in future release of vSphere.
   Workaround:
   
   • This process may need to be completed for multiple clusters. VMware HA can be disabled only if there are no virtual machines with VMware Fault Tolerance (FT) enabled. If there are virtual machines with VMware FT enabled in the cluster you are disabling, turn off VMware FT before disabling VMware HA. The process of turning off VMware FT is described in Turn Off VMware FT.
   • Virtual Machine Monitoring will also be disabled when HA is disabled.
   • The process may take some time to complete.

To disable vSphere HA in the vSphere Client:

1. Log in to the vSphere Client.
2. From the Home screen, click on Hosts and Clusters.
3. Browse to the cluster.
4. Click the Configure tab.
5. Select vSphere Availability and click Edit.
6. Deselect the VMware HA option.
7. Click OK.

To enable vSphere HA in the vSphere Client:

1. Log in to the vSphere Web Client
2. From the Home screen, click on Hosts and Clusters.
3. Browse to the cluster.
4. Click the Configure tab
5. Select vSphere Availability and click Edit.
6. Select the VMware HA option.
7. Click OK.

• •