Attempt to upgrade SDDC manager fails with task been cancelled and error "Deployment level lock cannot be acquired. There are existing resource locks"

Products

• VCF Operations
• VMware SDDC Manager

Issue/Introduction

• On SDDC manager Ui we are seeing the upgrade status changing to CANCELLED
  
  
• The SDDC manager logs show the following:
  
  /var/log/vmware/vcf/lcm/lcm-debug.log
yyyy-mm-ddThh:mm:ss.zzz+0000 INFO [vcf_lcm,<task_id>] [c.v.evo.sddc.lcm.orch.Orchestrator,Scheduled-10] Starting orchestrating scheduled upgrade <upgrade_id>
yyyy-mm-ddThh:mm:ss.zzz+0000 INFO [vcf_lcm,<task_id>] [c.v.evo.sddc.lcm.orch.Orchestrator,Scheduled-10] Attempting to acquire deployment lock for upgrade <upgrade_id>
yyyy-mm-ddThh:mm:ss.zzz+0000 DEBUG [vcf_lcm,<task_id>] [c.v.e.s.l.a.i.i.LogicalInventoryClient,Scheduled-10] logical inventory - acquire resource lock http://localhost/locks
yyyy-mm-ddThh:mm:ss.zzz+0000 INFO [vcf_lcm,<task_id>] [c.v.e.s.l.a.i.i.LogicalInventoryClient,Scheduled-10] acquired the resource lock for the resource:
{
    "status": "FAILED",
    "createdTimeStamp": 0,
    "errorMessage": "Deployment level lock cannot be acquired. There are existing resource locks.",
    "resourceType": "DEPLOYMENT"
}
yyyy-mm-ddThh:mm:ss.zzz+0000 WARN [vcf_lcm,<task_id>] [c.v.evo.sddc.lcm.orch.Orchestrator, Scheduled-10] Cannot start upgrades since there are pending or, failed workflows
yyyy-mm-ddThh:mm:ss.zzz+0000 INFO [vcf_lcm,<task_id>] [c.v.e.s.l.s.impl.UpgradeServiceImpl, Scheduled-10] Cancelling independent upgrade <upgrade_id>
yyyy-mm-ddThh:mm:ss.zzz+0000 INFO [vcf_lcm,<task_id>] [c.v.e.s.l.d.c.u.UpgradeClientImpl, Scheduled-10] Upgrade status is SCHEDULED, changing it to CANCELLED.

Environment

VCF Operations 9.x
SDDC manager 9.x
SDDC manager 5.x

Cause

A previous task failed and did not release the deployment lock

Resolution

To manually release the lock, please contact Broadcom Support.