vMotion tasks stuck on target vCenter with stale VMs that cannot be deleted during cross-vCenter vMotion

Products

• VMware vSphere ESXi

Issue/Introduction

• The vMotion tasks were cancelled on the source vCenter, but the corresponding vMotion tasks remain stuck on the target vCenter for a long time.
• The virtual machines (VMs) continue running normally on the source ESXi hosts and source vCenter.
• Stale virtual machines are observed on the target vCenter and target ESXi hosts, and they cannot be deleted even though they are powered off.
• When attempting to unregister the stale virtual machines on the target ESXi host using vim-cmd vmsvc/unregister <VM-ID>, the following error is observed:
  The attempted operation cannot be performed in the current state (Powered off)

Environment

VMware vCenter Server 8.0.x

VMware vSphere ESXi 8.0.x

Cause

The vMotion tasks remain in a hung state in the target vCenter, leaving the VM processes running on the target ESXi hosts and thereby preventing the deletion of the stale VMs.

Resolution

1. Cancel the stuck vCenter tasks by restarting the vCenter services  (Refer to Knowledge Base article Clear stale tasks from the vCenter "Recent Tasks" panel for details)

service-control --stop --all && service-control --start --all

2. Kill the stale VM processes on the target ESXi hosts (Refer to Knowledge Base article Unable to unregister virtual machine showing power off status in ESXi 8 for details)

esxcli vm process list | grep -B10 -ie '<VM-NAME>'

esxcli vm process kill --type=soft --world-id=<World ID>

3. Delete the stale VMs from the target ESXi hosts and target vCenter if they still exist

4. Initiate the cross-vCenter vMotion for the VMs again