Error: "Internal Server Error - Expected completed future, but received future which is still in progress" when changing VM placement policy or VM storage policy in Cloud Director 10.5.1.1

Products

• VMware Cloud Director

Issue/Introduction

• After upgrading to Cloud Director 10.5.1.1 or later, changing a VM placement policy or VM storage policy triggers a storage vMotion on the vSphere side, but in Cloud Director, after a few minutes the task fails.
• The task shows the following error message
  
  "PlacementException NO_FEASIBLE_PLACEMENT_SOLUTION"
• In the log file /opt/vmware/vcloud-director/logs/vcloud-container-debug.log the following is observed:
  

  ERROR    | task-service-activity-pool-1056 | FutureUtil                     | Failed to relocate VM to a Hub | requestId=<requestid>,request=POST https://cloud.example.com/api/vApp/vm-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/action/reconfigureVm

  DEBUG    | task-service-activity-pool-1056 | VAppManagerImpl                | 1st Attempt to update storage class <storage policy name>(com.vmware.vcloud.entity.vdcstorageProfile:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx) of VM <vm name>(com.vmware.vcloud.entity.vm:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx) failed | requestId=<requestid>,request=POST https://cloud.example.com/api/vApp/vm-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/action/reconfigureVm

  ERROR    | Backend-activity-pool-295668 | RelocateVmActivity             | [Activity Execution] Uncaught Exception during Activity execution. Recent phase: com.vmware.ssdc.backend.services.impl.RelocateVmActivity$UpdateComputeVmModelIfNeeded@6d242c8e - Handle: urn:uuid:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx, Current Phase: RelocateVmActivity$UpdateComputeVmModelIfNeeded | requestId=<requestid>,request=POST https://cloud.example.com/api/vApp/vm-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/action/reconfigureVm,requestTime=1713265903823,remoteAddress=<ipaddress>,userAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ...,accept=application/*+xml;version 39.0.0-alpha vcd=<uuid>,task=<taskid>activity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:<uuid>) activity=(com.vmware.ssdc.backend.services.impl.RelocateVmActivity,urn:uuid:<uuid>)
java.util.concurrent.ExecutionException: java.lang.AssertionError: Expected completed future, but received future which is still in progress
        at com.vmware.vcloud.activity.executors.ActivityRunner.run(ActivityRunner.java:124)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.AssertionError: Expected completed future, but received future which is still in progress

Environment

VMware Cloud Director 10.5.1.1 or later

Cause

vCenter will trigger a storage vMotion when the policy is updated in Cloud Director. This can take some time to complete.  In Cloud Director 10.5.1.1, there is new timeout setting introduced for which when a VM policy change or a Storage vMotion task takes longer than default 5 minutes, the task would fail at VCD level as the timeout is hit.

Resolution

To resolve this issue the relocate timeout property (relocate.vm.workflow.timeout.minutes)  is available and configurable in Cloud Director.

You will need to increase the relocate timeout value higher than the default 5 minutes using the following command.

/opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n relocate.vm.workflow.timeout.minutes -v 10

We recommend to increase this value by 5 minutes each time, until you find a value that allow the change of the VM placement policy. The value needs to be higher than the time take for the storage vMotion to complete in vSphere.

Note: The value is in minutes. The command can be run from any cell. No reboot or restart of service required.

To confirm the setting has been applied you can run the following command:

/opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n relocate.vm.workflow.timeout.minutes -l

 

NOTE: 

There is no max limit for the relocate.vm.workflow.timeout.minutes value. The value can be increased the as per your requirement and then if you wish you can set it back to default value.