Error: Password Manager operations working on related resources - SDDC Manager

Products

• VMware Cloud Foundation

Issue/Introduction

SDDC Manager tasks such as host commissioning, vCenter upgrades, or password rotations fail when a previous workflow leaves a stale resource lock in the internal database. This occurs when a task enters a terminal failed state without releasing the lock.

• When attempting to perform any password-related task in Password Manager, one or more of the following errors may be encountered:

  • “This operation is not allowed because Password Manager operation(s) are working on related resources.”

• SDDC Manager UI shows a red banner:

• • "The scheduled auto-rotate workflow could not run due to another operation blocking the resources."
  • “The operation is not allowed because the system lock is held by a Password Manager operation in progress.”

• In SDDC /var/log/vmware/vcf/operationsmanager/operationsmanager.log
yyyy-mm-ddThh:mm:ss DEBUG [vcf_om,40#######ef,f8fb] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7300-exec-4] Handler Error Response: {"errorCode":"PASSWORD_MANAGER_UPDATE_IN_PROGRESS","arguments":["d68f####-####-####-###99"],"message":"Password update task d68f####-####-####-###99 is in progress/failed state, retry after sometime.","remediationMessage":"Wait for currently in-progress transaction to complete as succeeded/cancelled.","referenceToken":"######"}
  
  
• Host commissioning fails with: "Password Manager operations are working on related resources."
  
  
• The Update Now button for vCenter or other bundles is greyed out.

Environment

• VCF 5.x
• VCF 9.x

Cause

A password management workflow (rotation or update) entered a FAILED or PREVALIDATION_FAILED state but did not release the database resource lock in the operationsmanager or platform databases.

Resolution

Take a snapshot of the SDDC Manager VM (without memory) before proceeding, then choose one of the options below:

1. Clear Failed Tasks via UI or API
   
   
   1. Navigate to Password Management in the SDDC Manager UI.
      
      
   2. Click Cancel on any failed task banners if available.
      
      
   3. If the UI option is unavailable, use the API Explorer:
      1. Navigate to Developer Center > API Explorer > APIs for Managing Credentials.
      2. Locate DELETE /v1/credentials/tasks/{id}.
      3. Enter the UUID of the failed task and click Execute.
         
         
         
2. Clear Stale Tasks via Database
   
   
   1. Establish an SSH into the SDDC Manager with vcf user and switch to root
      
      
   2. Identify failed tasks:
psql -h localhost -U postgres -d operationsmanager -c "select workflow_id, operation_type, transaction_status from passwordmanager.password_operations where transaction_status='FAILED' OR transaction_status='PREVALIDATION_FAILED';"
      
      
   3. Reset task status to cancelled:
      psql -h localhost -U postgres -d operationsmanager -c "update passwordmanager.password_operations set transaction_status='USER_CANCELLED' where transaction_status='FAILED' OR transaction_status='PREVALIDATION_FAILED';"
   
   
3. Clear Deployment Locks

1. 1. Check for active deployment locks:
      psql -h localhost -U postgres -d platform -c "select * from lock;"
      
      
   2. Delete the stale lock if the resource_type is deployment :
      psql -h localhost -U postgres -d platform -c "delete from lock where resource_type='deployment';"
      
      
   3. Restart SDDC Manager services:
      /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

For further assistance, see Contact Broadcom Support.

Additional Information

Multiple tasks in SDDC Manager fails with Error: ‘This operation is not allowed because SDDC Manager Backup Operation(s) are working on related resources.’