Incomplete vApp Template Replication due to Lock Acquisition Timeouts in VMware Cloud Director Availability

Products

• VMware Cloud Director

Issue/Introduction

When initiating a vApp Template Replication in VMware Cloud Director Availability (VCDA), the workflow finishes without showing explicit errors in the UI, but it fails to replicate all the virtual machines contained within the template.

• Example: A source template contains seven VMs, but only three VMs appear or are successfully processed on the destination site.

• The issue persists even if the source and destination environment configurations (such as vSAN storage profiles, thin provisioning, unique UUIDs, and account permissions) are validated and correct.

• Reviewing the /opt/vmware/h4/cloud/log/cloud.log on the replication appliance reveals a lock acquisition timeout and severe slowness during Runtime VM (RT VM) operations:

com.vmware.h4.manager.api.exceptions.LockTimeoutException: Lock acquisition timed out for object: 'C4VAPP-########-####-####-####-########'
    at com.vmware.h4.cloud.job.VmReplicationJob.lambda$persistRtVm$32(VmReplicationJob.java:972)

• You will also observe long delays when moving the RT VM into an existing vApp wrapper (sometimes taking upwards of 8 minutes per VM):

INFO - [...] [job-71] com.vmware.h4.cloud.service.RtVmService : Moving RT VM ########-####-####-####-######## into existing vApp ########-####-####-####-########

Environment

• VMware Cloud Director Availability 4.x
• VMware Cloud Director 10.x

Cause

VMware Cloud Director (VCD) does not natively support concurrent recompose operations. When a vApp template replication workflow starts, individual VM replications must sequentially acquire a lock on the destination parent vApp object.

If the underlying VCD environment experiences performance issues or slow RT VM management operations, the concurrent VM replications will queue up waiting for the vApp lock. By default, the lock acquisition timeout is set to 20 minutes. If a VM replication cannot obtain the lock within this 20-minute window, it fails silently in the UI, causing only the first few VMs (those that successfully acquired the lock in time) to be visible and replicated.

Resolution

Increase the C4 vApp Lock Timeout to  configuration to 60 minutes (3,600,000 milliseconds) by following the steps below

• SSH into the destination C4 replication appliance.
• Open the application properties configuration file for editing: /opt/vmware/h4/cloud/config/application.properties
• Add (or update, if it already exists) the following configuration line:

       default.lock.timeout=3600000

• Restart the cloud service to apply the new timeout window:

       systemctl restart cloud

• Delete the incomplete replication and reconfigure the vApp template replication completely from scratch.