SDDC Manager API Password Rotation fails with "UPDATE_ROTATE_PASSWORDS is not supported"

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

When attempting to rotate multiple passwords at the same time (in parallel) using PowerShell scripts or third-party orchestration tools via the SDDC Manager API, the operation fails with the following error:

RemoteException: {"errorCode":"Forbidden","message":"The attempted functionality UPDATE_ROTATE_PASSWORDS is not supported in the current system state. This operation is not allowed because Password Manager operation(s) are working on related resources."}

The issue occurs when multiple API sessions are initiated simultaneously, while individual rotations executed one-by-one succeed without error.

Cause

This is expected behavior in VMware Cloud Foundation (VCF). SDDC Manager places a system-level lock on the Password Manager service during any active credential task (Rotate, Update, or Remediate). Because of this lock, SDDC cannot process more than one request that affect related resources, and as a result, the "Forbidden" errorCode response is returned.

Resolution

To resolve this issue, password rotate API calls must be serialized. The task statuses can be monitored so that programatically the next rotate is called once the previous is completed. 

 

Alternatively, you can rotate multiple resources of the same type by selecting a group of resources in Password Management and then click "ROTATE NOW" to queue the password rotation tasks. You can also click the "ROTATE ALL" button to rotate all passwords of the currently selected resource type.