SDDC Manager password operations are not allowed because of a failed password task

Products

• VMware Cloud Foundation

Issue/Introduction

When attempting to perform any password task one of these errors is seen

"This operation is not allowed because Password Manager operation(s) are working on related resources" 

"The scheduled auto-rotate workflow could not run due to another operation blocking the resources" banner on VCF UI

"THE OPERATION IS NOT ALLOWED BECAUSE THE SYSTEM LOCK IS HELD BY PASSWORD MANAGER OPERATION IN PROGRESS" 

Similar errors are seen in the operationsmanager.log

/var/log/vmware/vcf/ioperationsmanager.log

yyyy-mm-ddThh:mm:ss DEBUG [vcf_om,404f8e66713d47ef,f8fb] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7300-exec-4] Handler Error Response: {"errorCode":"PASSWORD_MANAGER_UPDATE_IN_PROGRESS","arguments":["d68f530d-1cd7-4988-af5b-28f8dda87699"],"message":"Password update task d68f530d-1cd7-4988-af5b-28f8dda87699 is in progress/failed state, retry after sometime.","remediationMessage":"Wait for currently in-progress transaction to complete as succeeded/cancelled.","referenceToken":"GURKAF"}
 

Environment

VMware Cloud Foundation 3.11
VMware Cloud Foundation 4.x

VMware Cloud Foundation 5.1.1.0

Cause

• There might be a password rotation task failed on individual components due to other reasons(Password expired, SSH disabled etc)
• Reported failed workflow of password manager holds a lock due to task being in FAILED or IN_PROGRESS state 

Note: The task should either be in USER_CANCELLED or SUCCESSFUL state

Resolution

1. Take a snapshot(without memory) of SDDC Manager VM 
2. SSH into the SDDC Manager with vcf user and su to root
3. Run the following query to list the password operations in FAILED and PREVALIDATION_FAILED state
   • psql -h localhost -U postgres -d operationsmanager -c "select workflow_id, operation_type, transaction_status from passwordmanager.password_operations where transaction_status='FAILED' OR transaction_status='PREVALIDATION_FAILED';"
   • Sample output
     •             workflow_id              | operation_type |  transaction_status
--------------------------------------+----------------+----------------------
 xxxxxxxx-xxxx-xxxx-xxxx-28f8dda87699 | ROTATE         | PREVALIDATION_FAILED
4. Delete the password operation(s) task
   • Connect to the SDDC UI
   • Click on Developer Center > API Explorer > APIs for Managing Credentials
   • Go to DELETE/v1/credentials/tasks/{id}
   • Paste the id xxxxxxxx-xxxx-xxxx-xxxx-28f8dda87699 ----> identified in Step # 3
   • 
   • Click EXECUTE
5. Re-run the postgres query in Step # 3 and ensure there are no more failed password operations. 
6. Refresh the SDDC Manager UI
7. If still unable to rotate password, restart the services via an SSH session and running:
   /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

Additional Information

Impact/Risks:
None: The resolution will remove the locks instilled due to failed task. A snapshot of SDDC manager without memory is still recommended