Password Manager Operations are Blocked by Stale Resource Locks

Products

• VMware Cloud Foundation
• VMware SDDC Manager / VCF Installer

Issue/Introduction

SDDC Manager tasks such as host commissioning, vCenter upgrades, or password rotations may fail when a previous workflow leaves a stale resource lock in the internal database. This occurs when a task enters a terminal failed state without releasing the lock and some of errors encountered as listed beow:

• When attempting to perform any password-related task in Password Manager, one or more of the following errors may be encountered:
  This operation is not allowed because Password Manager operation(s) are working on related resources.
  
  
• SDDC Manager UI may show a red banner:
  The scheduled auto-rotate workflow could not run due to another operation blocking the resources.

The operation is not allowed because the system lock is held by a Password Manager operation in progress.

• In SDDC Manager /var/log/vmware/vcf/operationsmanager/operationsmanager.log entries similar to below are observed:
  
  YYYY-MM-DDThh:mm:ss DEBUG [vcf_om,########,####] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7300-exec-4] Handler Error Response: {"errorCode":"PASSWORD_MANAGER_UPDATE_IN_PROGRESS","arguments":["####-####-####-###"],"message":"Password update task ####-####-####-### is in progress/failed state, retry after sometime.","remediationMessage":"Wait for currently in-progress transaction to complete as succeeded/cancelled.","referenceToken":"######"}
  
  
• Host commissioning tasks from SDDC Manager may fail with below error:
  Password Manager operations are working on related resources.
  
  
• Attempting to remove host from cluster using SDDC Manager shows task being queued due to domain lock being unavailable on the /var/log/vmware/vcf/domainmanager/domainmanager.log

YYYY-MM-DDThh:mm:ss DEBUG [vcf_dm,########,####] [c.v.v.q.s.QueueProcessingService,dm-scheduler-3] REMOVE_HOST_WORKFLOW task ####-####-####-#### remains in queue because resource lock domain is unavailable
  
  
• The 'Update Now' button for vCenter or other bundles might be greyed out.

Environment

SDDC Manager 5.X
VCF 9.X

Cause

A password management workflow (rotation or update) entered a FAILED or PREVALIDATION_FAILED state but did not release the database resource lock in the operationsmanager or platform databases.

Resolution

Take a snapshot of the SDDC Manager VM (without memory) before proceeding, then choose one of the options below:

1. Clear Failed Tasks via UI or API
   
   
   1. Navigate to Password Management in the SDDC Manager UI.
      
      
   2. Click Cancel on any failed task banners if available.
      
      
   3. If the UI option is unavailable, use the API Explorer:
      1. Navigate to Developer Center > API Explorer > APIs for Managing Credentials.
      2. Locate DELETE /v1/credentials/tasks/{id}.
      3. Enter the UUID of the failed task and click Execute.
         
         
         
         
2. Clear Stale Tasks via Database
   
   
   1. Establish an SSH into the SDDC Manager with vcf user and switch to root
      
      
   2. Identify failed tasks:
psql -h localhost -U postgres -d operationsmanager -c "select workflow_id, operation_type, transaction_status from passwordmanager.password_operations where transaction_status='FAILED' OR transaction_status='PREVALIDATION_FAILED';"
      
      
   3. Reset task status to cancelled:
      psql -h localhost -U postgres -d operationsmanager -c "update passwordmanager.password_operations set transaction_status='USER_CANCELLED' where transaction_status='FAILED' OR transaction_status='PREVALIDATION_FAILED';"
   
   
3. Clear Deployment Locks

1. 1. Check for active deployment locks:
      psql -h localhost -U postgres -d platform -c "select * from lock;"
      
      
   2. Delete the stale lock if the resource_type is deployment :
      psql -h localhost -U postgres -d platform -c "delete from lock where resource_type='deployment';"
      
      
   3. Restart SDDC Manager services:
      /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

For further assistance, see Contact Broadcom Support.

Additional Information

Multiple tasks in SDDC Manager fails with Error: ‘This operation is not allowed because SDDC Manager Backup Operation(s) are working on related resources.’