Upgrading components via SDDC manager fail with error "System initiated upgrade cancel operation due to Cannot start upgrades since locks could not be acquired due to exception"

Products

• VCF Operations
• VMware SDDC Manager / VCF Installer

Issue/Introduction

• In the SDDC Manager UI, the upgrade task for any component [vCenter, NSX, ESXi, SDDC, etc] fails with the following error:
  
System initiated upgrade cancel operation due to Cannot start upgrades since locks could not
be acquired due to exception
  
  
• In /var/log/vmware/vcf/lcm/lcm-debug.log:

yyyy-mm-ddThh:mm:ss.zzz+0000 INFO [vcf_lcm,<task_id>] [c.v.evo.sddc.lcm.orch.Orchestrator,Scheduled-10] Starting orchestrating scheduled upgrade <upgrade_id>
yyyy-mm-ddThh:mm:ss.zzz+0000 INFO [vcf_lcm,<task_id>] [c.v.evo.sddc.lcm.orch.Orchestrator,Scheduled-10] Attempting to acquire deployment lock for upgrade <upgrade_id>
yyyy-mm-ddThh:mm:ss.zzz+0000 DEBUG [vcf_lcm,<task_id>] [c.v.e.s.l.a.i.i.LogicalInventoryClient,Scheduled-10] logical inventory - acquire resource lock http://localhost/locks
yyyy-mm-ddThh:mm:ss.zzz+0000 INFO [vcf_lcm,<task_id>] [c.v.e.s.l.a.i.i.LogicalInventoryClient,Scheduled-10] acquired the resource lock for the resource:
{
    "status": "FAILED",
    "createdTimeStamp": 0,
    "errorMessage": "Deployment level lock cannot be acquired. There are existing resource locks.",
    "resourceType": "DEPLOYMENT"
}
yyyy-mm-ddThh:mm:ss.zzz+0000 WARN [vcf_lcm,<task_id>] [c.v.evo.sddc.lcm.orch.Orchestrator, Scheduled-10] Cannot start upgrades since there are pending or, failed workflows
yyyy-mm-ddThh:mm:ss.zzz+0000 INFO [vcf_lcm,<task_id>] [c.v.e.s.l.s.impl.UpgradeServiceImpl, Scheduled-10] Cancelling independent upgrade <upgrade_id>
yyyy-mm-ddThh:mm:ss.zzz+0000 INFO [vcf_lcm,<task_id>] [c.v.e.s.l.d.c.u.UpgradeClientImpl, Scheduled-10] Upgrade status is SCHEDULED, changing it to CANCELLED.



• Executing the following command against the platform database confirms the presence of multiple orphaned locks:

  psql -h localhost -U postgres -d platform -c "select * from lock;"

In the output, the column 'locking_context' displays the same operationIdentifier for all the entries indicating that all the locks are due to the same stale task.

Environment

VCF Operations 9.x
SDDC Manager 9.x
SDDC Manager 5.x

Cause

A previous configuration task or automated system workflow terminated or stalled unexpectedly.

This interruption left an orphaned operational lock row on the specified resource mapping inside the platform inventory database.

Resolution

1. Take an offline snapshot of the SDDC Manager Virtual Machine.
   
   
2. Delete the orphaned lock strings from the Postgres platform database using the command below:
   
   psql -h localhost -U postgres -d platform -c "delete from lock where locking_context like '%<stale_operationIdentifier>%';"