Multiple tasks in SDDC Manager fails with Error: ‘This operation is not allowed because SDDC Manager Backup Operation(s) are working on related resources.’

Products

• VMware Cloud Foundation

Issue/Introduction

When attempting to perform domain-related operations in SDDC Manager such as host commissioning or VI domain creation or attempting to Remediate and Update Password you may encounter the following error message:

"This operation is not allowed because SDDC Manager Backup Operation operation(s) are working on related resources."

• SDDC Manager Backup is failing with the error: 'SDDC Manager Deployment lock to perform Backup & Restore Operation'.

The above instances indicate that a deployment lock is preventing further actions within the environment.

Environment

Vmware Cloud Foundation 4.5
VMware Cloud Foundation 5.0

Cause

This issue occurs when a restore operation in SDDC Manager fails to release a deployment lock that was previously acquired by a backup operation. As a result, any new deployment-related operations are blocked due to the lingering lock entry.

Resolution

This issue has been permanently resolved in VMware Cloud Foundation (VCF) 5.0. Upgrading to this version or later eliminates the lock persistence issue caused by restore operations.

Workaround:
If you are using a version of VCF prior to 5.0, the deployment lock can be manually removed by following these steps:

Important: Always take a snapshot of the SDDC Manager appliance before performing any database-level modifications.

1. SSH into the SDDC Manager appliance as the root user.
2. Validate a lock exists. 
   • psql -h localhost -U postgres -d platform -c "select * from lock;"
3. Delete the DB lock entry:
   • psql -h localhost -U postgres -d platform -c "delete from lock where resource_type='deployment';"

After performing these steps, domain-related operations in SDDC Manager should proceed without issue.

Note: If the issue persists for Remediate or Update Password operations, it's likely due to failed password tasks in the database.

• Refer to the following KB article for instructions on clearing failed password operations:  KB- SDDC Manager password operations are not allowed because of a failed password task