VM remains in source Organization vDC in Cloud Director after long-running vMotion or Storage vMotion

Products

• VMware Cloud Director

Issue/Introduction

• A Virtual Machine (VM) is displayed in the source Organization vDC within the VMware Cloud Director (VCD) UI, but it is physically located on the destination vCenter or cluster in the vSphere UI.
• A vMotion or Storage vMotion task was initiated via Cloud Director.
• The migration task took longer than 5 minutes to complete in vCenter.
• Cloud Director reports that the task failed or timed out, yet the VM move was completed successfully in vCenter.

Environment

• VMware Cloud Director 10.6.1
• vCenter Server

Cause

This issue is caused by the storage vMotion operation exceeding the default relocate.vm.workflow.timeout.minutes setting in Cloud Director, which is set to 5 minutes by default. When this timeout is reached, Cloud Director reverts the operation in its database to maintain consistency, but it does not cancel the task in vCenter. vCenter continues the operation until completion, leading to an inventory mismatch where VCD expects the VM in its original location.

Resolution

Part 1: Increase the Relocate Timeout

To prevent future occurrences for long-running migrations, increase the timeout value:

1. Log in to the Cloud Director primary cell via SSH.
2. Run the following command to increase the timeout to 60 minutes (or a value appropriate for your environment): /opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n relocate.vm.workflow.timeout.minutes -v 60
3. Verify the setting: /opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n relocate.vm.workflow.timeout.minutes -l

Part 2: Manual Inventory Alignment

To correct the current mismatched VM:

1. Power down the VM in vCenter.
2. Remove from Inventory: Right-click the VM in vCenter and select Remove from Inventory (do not select Delete from Disk).
3. Delete Phantom Entry: In the Cloud Director UI, attempt to delete the VM from the old vDC.
4. Re-register VM: In vCenter, browse the datastore where the VM resides, right-click the .vmx file, and select Register VM to add it to the correct destination vCenter/Cluster.
5. Import to VCD: Use the Import from vCenter option in the Cloud Director destination vDC to bring the VM back into the VCD inventory in the correct location.