Unable to perform password operations on SDDC Manager with error: "Failed to update the password. Password update task is in progress/failed state, retry after sometime"

Products

• VMware Cloud Foundation
• VMware SDDC Manager
• VMware Cloud Foundation 4.x
• VMware Cloud Foundation 5.x

Issue/Introduction

The purpose of this document is to resolve the error with the password operations, and allow successful credential operations ( Rotate / Update / Remediate ) in Password Manager Tab of the SDDC Manager UI.

When attempting to rotate a password (or perform any other password operation) in the SDDC Manager UI, there is a red banner that shows the error:

"Failed to update the password. Password update task <workflow_id> is in progress/failed state, retry after sometime."

In the operationsmanager.log, we see errors reporting the following:

YYYY-MM-DD HH:MM:SS+0000 DEBUG [vcf_om,############,####] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7300-exec-5] 
Handler Error Response: {"errorCode":"PASSWORD_MANAGER_UPDATE_IN_PROGRESS","arguments":["########-####-####-####-##############"],
"message":"Password update task ########-####-####-####-############## is in progress/failed state, retry after sometime.",
"remediationMessage":"Wait for currently in-progress transaction to complete as succeeded/cancelled.","referenceToken":"356O5Q"}

Environment

VMware Cloud Foundation 9.x
VMware Cloud Foundation 5.x
VMware Cloud Foundation 4.x
VMware Cloud Foundation 3.x

Cause

This is due to the reported workflow of a password operation being stuck, with its status as " IN_PROGRESS" or "FAILED" in the passwordmanager table of the operationsmanager DB.

Resolution

1. (Important) Take a snapshot of the SDDC Manager VM.
2. Connect to the SDDC Manager via SSH with the user vcf and then su root.
3. Using the workflow_id from the red banner error message in the UI (or from the error in operationsmanager.log), run the following command to check its current status:
   

   psql -U postgres -h localhost -d operationsmanager -c "select workflow_id, operation_type, transaction_status from passwordmanager.password_operations where workflow_id='<workflow_id>';"

   If we see the transaction_status as anything other than "SUCCEEDED" or "USER_CANCELLED" (such as "IN_PROGRESS , PREVALIDATION_FAILED, FAILED"), we need to change it status to "USER_CANCELLED".
   
   Note:
   If you are unable to get the workflow_id

   psql -U postgres -h localhost -d operationsmanager -c "select workflow_id, operation_type, transaction_status from passwordmanager.password_operations where transaction_status!='SUCCEEDED' or transaction_status!='USER_CANCELLED';"

    

4. Change the status of the  workflow_id using the following command:

   psql -U postgres -h localhost -d operationsmanager -c "update passwordmanager.password_operations set transaction_status='USER_CANCELLED' where workflow_id='<workflow_id>';"

   Note:
   To update the status to "USER_CANCELLED" without workflow_id.

   Warning: This will edit ALL password transaction operations that are not in a SUCCEEDED or USER_CANCELLED state. Please proceed with caution.

   psql -U postgres -h localhost -d operationsmanager -c "update passwordmanager.password_operations set transaction_status='USER_CANCELLED' where transaction_status !='SUCCEEDED' AND transaction_status !='USER_CANCELLED';"

5. Restart the SDDC Manager services:

   /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

6. Attempt the password operation again from the SDDC Manager UI.

 

Additional Information

You can run the following query to see all password operations that have a transaction_status that does not equal SUCCEEDED and USER_CANCELLED.

psql -U postgres -h localhost -d operationsmanager -c "select workflow_id, operation_type, transaction_status from passwordmanager.password_operations where transaction_status !='SUCCEEDED' AND transaction_status !='FAILED';"

Impact/Risks:
MODERATE: The resolution steps involve editing database entries on the SDDC Manager and restarting all services.
A snapshot of the SDDC Manager is required as a fallback.