Error: "Inventory timed out waiting for enable relocateVM_Task for VM..." when failing over vApp Replications

Products

• VMware Cloud Director

Issue/Introduction

• When failing over vApp replications in VMware Cloud Director Availability (VCDA), the failover times out for some of the VM replications and you see errors similar to:
  
  Unexpected VMware Cloud Director error. [ UI-########-####-####-####-########cb3e-r1661-38-JP-3K8-########-####-####-####-########c3b0 ] Inventory timed out waiting for enable relocateVM_Task for VM [vcId=########-####-####-####-########8cdf, moref=vm-###]
  
  
• In /opt/vmware/vcloud-director/logs/vcloud-container-debug.log on the VMware Cloud Director (VCD) cell that performs a post failover reconfigure task on the recovered VM, you see entries similar to:
  
  2025-06-19 09:32:15,193 | ERROR    | Backend-activity-pool-98330 | RelocateVmActivity             | An exception occurred while relocating to destination hub moref://########-####-####-####-########28cdf/ResourcePool#resgroup-### attempting to reassign vm to source hub urn:uuid:########-####-####-####-########cd24, and undo destination hub moref://########-####-####-####-########8cdf/ResourcePool#resgroup-### reconfiguration if one was performed | requestId=UI-########-####-####-####-########cb3e-r1661-38-JP-3K8-########-####-####-####-########c3b0,request=PUT https://<VCD FQDN>/api/vApp/vm-########-####-####-####-########f664/networkConnectionSection,requestTime=1750325230190,remoteAddress=##.##.##.##:###,userAgent=Apache-HttpClient/4.5.14 (Java/17.0.12),accept=application/*+xml;version 38.0 vcd=########-####-####-####-########9e78,task=########-####-####-####-########55df activity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:########-####-####-####-########55df) activity=(com.vmware.ssdc.backend.services.impl.RelocateVmActivity,urn:uuid:########-####-####-####-########28f6)

java.util.concurrent.TimeoutException: Inventory timed out waiting for enable relocateVM_Task for VM [vcId=########-####-####-####-########8cdf, moref=vm-#####]
        at com.vmware.ssdc.backend.services.impl.RelocateVmActivity$RelocateVmAcrossRpAndDsPhase.invoke(RelocateVmActivity.java:838)
        at com.vmware.vcloud.activity.executors.ActivityRunner.runPhase(ActivityRunner.java:175)
        ...
  
  
• This issue can occur for vApp replications with a large number of instances per VM replication and consolidation enabled.

Environment

VMware Cloud Director Availability 4.7.x

Cause

This issue occurs when the consolidation of the VM replication instances take a long time to complete, causing the task to enable the relocate VM method for the recovered VM to stall and time out in Cloud Director after 5 minutes.

Resolution

This is a known issue affecting VMware Cloud Director Availability.

Subscribe to this knowledge article to get updates on this issue.

Workaround

Performing consolidation of instances can have a potential large increase in the Recovery Time Objective (RTO) of a failover depending on the following criteria:

• The number of disks per recovered VM.
• The size of each of the VM's disks.
• The data change rate of the VM.
• The number of instances retained for the VM.

To work around this issue, leave the Instances handling after recovery setting at Default so as to not consolidate the VMs on failover. The instances can then be manually consolidated after the recover task completes.

After performing the above, if issues still persist, then you can also increase the relocate timeout value in Cloud Director. 

1. Connect to any Cloud Director cell in the server group and login as the root user.
2. You can increase the relocate timeout value higher than the default 5 minutes using the following command. The value needs to be higher than the time take for the storage vMotion to complete in vSphere.
   This example shows a setting of 10 minutes.
   
   /opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n relocate.vm.workflow.timeout.minutes -v 10
   
   Note: The value is in minutes. The command can be run from any cell. No reboot or restart of service required.
   
   
3. To confirm the setting has been applied you can run the following command:
   
   /opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n relocate.vm.workflow.timeout.minutes -l

Note: There is no max limit for the relocate.vm.workflow.timeout.minutes value. The value can be increased the as per your requirement and then if you wish you can set it back to default value.