Synchronize inventory versions task fails to acquire lock in a VMware Cloud Foundation Environment

Products

• VMware SDDC Manager

Issue/Introduction

When scheduled daily job to synchronize the inventory versions fails to acquire lock, due to other parallel operations currently running, a task failure would be observed in the VMware Cloud Foundation task panel.

 

Environment

VCF 5.2

Cause

Synchronize inventory versions workflow failed to acquire lock. The Task "Acquire Lock" would fail, with error message saying "Lock is unavailable" and remediation to try again later. This can happen when another parallel workflow has already obtained the inventory lock, when the scheduled daily job triggers.

Resolution

• Manually restart the failed task in the task panel after the other parallel operations complete.
• Alternatively, wait for the daily version synchronization to synchronize the inventory versions on the next scheduled run.

• Workaround:
  If you are using a version of VCF prior to 5.0, the deployment lock can be manually removed by following these steps:

  Important: Always take a snapshot of the SDDC Manager appliance before performing any database-level modifications.

  1. SSH into the SDDC Manager appliance as the root user.
  2. Validate a lock exists. 
     • psql -h localhost -U postgres -d platform -c "select * from lock;"
  3. Delete the DB lock entry:
     • psql -h localhost -U postgres -d platform -c "delete from lock where resource_type='deployment';"

  After performing these steps, domain-related operations in SDDC Manager should proceed without issue.