Cloud Service Broker Upgrade All Services Errand fails with Error: creating Secrets Manager Secret Rotation

Products

• VMware Tanzu Application Service

Issue/Introduction

When upgrading Cloud Service Broker (CBS), the upgrade fails with the rotating secrets, a sample error is as follows:

2025-11-11T11:00:06Z: upgrade of instance: "postgres-shared-csb" guid: "#########" failed after 2m10.572320105s: upgrade failed: Error: creating Secrets Manager Secret Rotation (): operation error Secrets Manager: RotateSecret, https response error StatusCode: 400, RequestID: ebe70686-#####, api error ValidationException: Invalid name. Must be a valid name containing alphanumeric characters, or any of the following: -/_+=.@! with aws_secretsmanager_secret_rotation.secret_manager[0], on main.tf line 101, in resource "aws_secretsmanager_secret_rotation" "secret_manager": 101: resource "aws_secretsmanager_secret_rotation" "secret_manager" { exit status 1  

Cause

The error is caused by an error returned by an AWS API endpoint. A request Id id included in the error message

ebe70686-###### api error ValidationException.

If enabled more information or details can be taken from CloudWatch filtering log with the specific request id from the error message, or you can follow up with AWS Support which might help you give more info on what is causing the API error

Resolution

Follow the steps below to workaround this issue.

1.) Restore a database with a command like:

aws rds restore-db-instance-from-db-snapshot --db-snapshot-identifier snapshot-restore-test-db --db-instance-identifier csb-postgresql-######--db-instance-class db.t3.micro --db-subnet-group-name csb-postgresql-#######-p-sn --no-publicly-accessible --vpc-security-group-ids sg-#######

Note: that there is a --manage-master-user-password flag available, but it is currently documented as only working for Oracle DB.

2.) Update the instance to use a managed admin password:

aws rds modify-db-instance --db-instance-identifier csb-postgresql-####### --manage-master-user-password

3.) Run an update on the service to synchronize the state:

cf update-service csb-aws-postgresql-indigo-raven --wait -c '{}'

This step is necessary because some aspects of the new instance will be different - for example the ARN of the managed secret will likely be different, and the CSB needs to read the new value in order to bind/unbind apps.

4.) Continue with the CSB upgrade. Apply Change with the Upgrade All Service Errand.

Notes:

•  AWS CLI is used to re-enable the managed admin password, rather than using the Cloud Service Broker,  this would be advantageous to the CSB administrator as it means that the plan would not need to be modified.
• It should also be noted that some of the commands above take some time to apply - for example the command to enable the managed password will exit before the action is complete. It's important to wait for the actions to complete before performing the next step, otherwise they will fail.