Error: "Lock acquisition timed out for object: 'C4VAPP-<uuid>'" when configuring vApp or vApp template replications

Products

• VMware Cloud Director

Issue/Introduction

• When you configure or reverse a vApp or vApp template replication containing a large number of VMs in VMware Cloud Director Availability (VCDA), the task fails and you see an error similar to:
  
  Lock acquisition timed out for object: 'C4VAPP=<uuid>'
  
  
• In the /opt/vmware/h4/cloud/log/cloud.log file on the destination Cloud Director Replication Management Appliance, you see entries similar to:
  
  2024-08-17 15:18:10.900 DEBUG - [be70224f-bcef-4642-8f30-3baf0a9436b2] [c4-scheduler-1] c.vmware.h4.jobengine.lock.LockManager   : Expiring candidate c7ce1f8b-7f35-4a51-9b84-2aacad6926f1 for lock vapp_C4VAPP-<uuid>
2024-08-17 15:18:10.900 DEBUG - [UI-e4467c13-ba56-4349-bde8-999c9357a662-r665-QA-iep] [c4-scheduler-1] c.vmware.h4.cloud.job.VmReplicationJob   : Failed to acquire lock for RT vApp creation.
2024-08-17 15:18:10.900 ERROR - [UI-e4467c13-ba56-4349-bde8-999c9357a662-r665-QA-iep] [c4-scheduler-1] com.vmware.h4.jobengine.JobExecution     : Task c7ce1f8b-7f35-4a51-9b84-2aacad6926f1 (WorkflowInfo{type='start', resourceType='vmReplication', resourceId='C4-<uuid>', isPrivate=false, resourceName='null'}) has failed

com.vmware.h4.manager.api.exceptions.LockTimeoutException: Lock acquisition timed out for object: 'C4VAPP-<uuid>'
        at com.vmware.h4.cloud.job.VmReplicationJob.lambda$persistRtVm$29(VmReplicationJob.java:931)
        at com.vmware.h4.jobengine.lock.JobLock.lambda$lock$1(JobLock.java:93)
        at com.vmware.h4.jobengine.lock.LockManager.invokeHandler(LockManager.java:288)
        ...

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware Cloud Director Availability 4.x

Cause

This issue occurs because the default timeout value for a lock in VMware Cloud Director Availability is 10 minutes. When a vApp is configured to be replicated each VM replication task that is spawned is required to own the vApp replication lock when performing necessary configuration tasks. For large vApps, it is possible for this lock timeout to expire before all VM replications are processed, resulting in this error.

Resolution

To resolve this issue, increase the default timeout value based on the largest vApp you wish to replication and the time observed that it takes for individual VM replications to configure.

Note: This procedure modifies a configuration file. Ensure to take a backup of the file before proceeding.

1. SSH to the destination Cloud Director Replication Management Appliance and log in as root.
2. Navigate to the Cloud Serivce's application.properties file:
   
   cd /opt/vmware/h4/cloud/config/
   
   
3. Take a backup of the application.properties file:
   
   cp application.properties application.properties.<YYYY-MM-DD>.bak
   
   
4. Edit the application properties file with a text editor such as vi:
   
   vi application.properties
   
   
5. Append the file with the following line, adjusting the timeout value to suit your specific environment:
   
   default.lock.timeout=######
   
   Note: The timeout value is in milliseconds (ms) and the default value is 600000. Change the placeholder ###### to a new value based on your environmental needs.
   
   Example: If the largest vApp you wish to replication consists of 20 VMs and each VM replication takes ~45 seconds to configure, doubling the timeout to 20 minutes or 1200000 ms should be sufficient.
   default.lock.timeout=1200000
   
   
6. Save the changes to the file and restart the Cloud Service:
   
   systemctl restart cloud.service