Password remediation fails in SDDC Manager UI with error "Unable to acquire resource level lock(s) for workflow id ########"

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• Password remediation fails in SDDC Manager UI with below error message: 
  
  
  
  
• /var/log/vmware/vcf/operationsmanager/operationsmanager.log in SDDC appliance shows below events: 

YYYY-MM-DDTHH:MM:SS.SS1Z ERROR [vcf_om,#####-#####-#####-#####-#####] [c.v.v.o.s.l.LockingService, http-nio-#####-#####-exec-5] Workflow execution failed due to lock contention.
com.vmware.vcf.operationsmanager.exceptions.ResourceLockingException: Message: Unable to acquire resource level lock(s) for workflow id #####-#####-#####-#####-#####. Remediation Message: Check if locking APIs are up and responding. Reference Token: ######
 at com.vmware.vcf.operationsmanager.workflow.Engine.acquireLocks(Engine.java:452)
 ...
Caused by: com.vmware.vcf.operationsmanager.exceptions.LockAcquisitionException: Cause: Error in acquiring resource locks Lock for resource type domain cannot be acquired by OPERATIONS_MANAGER service and task. Reason: Resource is already locked by OPERATIONS_MANAGER service and task

Environment

VCF 5.x

Cause

This issue occurs because of a previous task or workflow has left a stale lock on the specified domain resource within the VMware Cloud Foundation (VCF) Operations Manager service preventing subsequent workflows or inventory synchronization tasks from acquiring the necessary resource-level locks. 

To resolve the stale resource lock causing password remediation failure in SDDC Manager, all services within the SDDC Manager appliance must be restarted. This refresh specifically clears temporary lock entries in services such as operationsmanager and commonsvcs.

Resolution

Proceed with the below steps to resolve the issue:

1. Log in to the SDDC Manager appliance via SSH using the vcf user account.

2. Switch to the root user: su -

3. Clear the stale resource lock by restarting all services in the SDDC Manager. Execute the following command to achieve it: /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

4. Wait for all services to initialize completely, then retry the failed workflow.

Additional Information

NOTE: During the restart, services like operationsmanager and commonsvcs refresh their state, which often clears temporary lock entries that were not properly cleaned up by the failed workflow. It terminates active service threads that might be incorrectly holding a "busy" status for that resource.