Tasks are stuck at 28% within Cloud Director

Products

• VMware Cloud Director

Issue/Introduction

• VM deployment tasks become stuck at 28% for a long time.
• Changing a VM placement policy on an existing VM results in a task stuck at 28%.
• The VM are configured to use placement policies.
• The Tenant uses elastic vDC which are backed by multiple vSphere compute clusters.
• Within the vCenter Server UI, cluster reconfigure tasks appear and shows 0% task status.
• A ReconfigureVmGroupActivity activity was the last action visible for the task and VMs remain in the pendingVmValRefList array in /opt/vmware/vcloud-director/logs/vcloud-container-debug.log

YYY-MM-DD 12:12:25,102 | DEBUG    | vim-proxy-activity-pool-31 | ReconfigureVmGroupActivity     | [Activity Execution] VMs [[vcId=########-####-####-#####-########, moref=vm-###]] are associated with vmGroup <Placement Group Name> in cluster [vcId=########-####-####-#####-########, moref=domain-#####] with operation ADD_VMS - Handle: urn:uuid:########-####-####-#####-########, Current Phase: ReconfigureVmGroupActivity$CreateGroupSpecsPhase | requestId=########-####-####-#####-########,request=POST https://cloud.example.com/api/vApp/vapp-########-####-####-#####-########/action/recomposeVApp,requestTime=1767706094713,remoteAddress=###.###.##.##:17226,userAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ...,accept=application/*+xml;version 40.0.0-alpha vcd=########-####-####-#####-########,task=########-####-####-#####-########activity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:########-####-####-#####-########) activity=(com.vmware.vcloud.vdc.impl.ComposeRecomposeVAppActivity,urn:uuid:########-####-####-#####-########) activity=(com.vmware.vcloud.vdc.impl.CreateStoredVmActivity,urn:uuid:########-####-####-#####-########) activity=(com.vmware.vcloud.fabric.storage.storedVm.impl.CreateFromExistingStoredVmNonBlockingActivity,urn:uuid:########-####-####-#####-########) activity=(com.vmware.vcloud.val.internal.impl.ReconfigureVmGroupActivity,urn:uuid:########-####-####-#####-########)
...
...
pendingVmValRefList=[[vcId=########-####-####-#####-########, moref=vm-###], [vcId=########-####-####-#####-########, moref=vm-###], [vcId=########-####-####-#####-########, moref=vm-###]], version=12] - Handle: urn:uuid:########-####-####-#####-########, Current Phase: ReconfigureVmGroupActivity$CreateGroupSpecsPhaseand VMs remain in the pendingVmValRefList

Environment

VMware Cloud Director 10.6.1

Cause

This issue occurs if there is a cross-cluster vMotion/Storage vMotion task in progress in vCenter Server. This activity creates a statelock (in vCenter Server) against both source and destination compute clusters. This statelock remains in place until the vMotion/Storage vMotion is complete. Subsequently any new task which needs to perform a cluster reconfigure operation will not be able to progress until that first task has completed. 

Cloud Director placement policy are mapped to VM/Host Groups objects which are created on the compute cluster in vCenter Server. When Cloud Director is configuring a VM to use a placement policy, it needs to perform a reconfigure task on the vSphere compute cluster. In this scenario it's unable to perform this reconfigure action as the vSphere compute cluster is in statelock due to the ongoing vMotion/Storage vMotion. This results in the reconfigure task becoming queued and stalling at 28% in Cloud Director.

Resolution

This is an expected vCenter Server behavior. For additional information see KB: Cluster Reconfigure operation stalls when a cross-cluster vMotion is in progress

Tasks which are queued should resume again as soon as the cross-cluster vMotion/Storage vMotion completes in vCenter Server.

Workaround

For new VM deployments:

1. Create the VM without any placement policy defined. When a placement policy is not defined, a cluster reconfigure task at the vSphere compute cluster level is not required.
2. As soon as the cross-cluster vMotion/Storage vMotion task completes, update the newly created VM to have a placement policy defined.