Multiple tasks in SDDC Manager fails with Error: ‘This operation is not allowed because SDDC Manager Backup Operation(s) are working on related resources.’

Products

• VMware Cloud Foundation

Issue/Introduction

When attempting to perform domain-related operations in SDDC Manager such as host commissioning or VI domain creation or attempting to Remediate and Update Password you may encounter the following error message:

1. "This operation is not allowed because SDDC Manager Backup Operation operation(s) are working on related resources."

2. SDDC Manager Backup is failing with the error: 'SDDC Manager Deployment lock to perform Backup & Restore Operation'.

The above instances indicate that a deployment lock is preventing further actions within the environment.

3. In some instances the "Commission hosts" option also gets greyed out in the SDDC manager UI :

Environment

Vmware Cloud Foundation 4.5
VMware Cloud Foundation 5.0
VMware Cloud Foundation 9.0

Cause

This issue occurs when a restore operation in SDDC Manager fails to release a deployment lock that was previously acquired by a backup operation. As a result, any new deployment-related operations are blocked due to the lingering lock entry.

Resolution

Workaround: Remove Deployment Lock

Important: Always take a snapshot of the SDDC Manager appliance before performing any database-level modifications.

1. SSH into the SDDC Manager appliance as the vcf user and then switch to root using : su -
2. Validate a lock exists. 
   • psql -h localhost -U postgres -d platform -c "select * from lock;"
3. Delete the DB lock entry:
   • psql -h localhost -U postgres -d platform -c "delete from lock where resource_type='deployment';"

After performing these steps, domain-related operations in SDDC Manager should proceed without issue.

Note: If the issue persists for Remediate or Update Password operations, it's likely due to failed password tasks in the database.

• Refer to the following KB article for instructions on clearing failed password operations:  KB- SDDC Manager password operations are not allowed because of a failed password task