General guideline about restoring RDS DB instance with enabling admin password managed by AWS Secrets Manager

Products

• VMware Tanzu Application Service

Issue/Introduction

The following feature is introduced in Tanzu Cloud Service broker (CSB) for AWS v1.13.0.

Enabling AWS Secrets Manager for admin password on MySQL, PostgreSQL and MSSQL - newly introduced properties use_managed_admin_password and rotate_admin_password_after enable the admin password for the RDS to be generated and managed by AWS Secrets Manager, including automatic rotation.

Sometimes the end user needs to restore a RDS DB instance form a DB snapshot and it will become a little complicated when the original RDS DB instance has set to have the admin password for the RDS to be generated and managed by AWS Secrets Manager.

Environment

Cloud Service broker for AWS v1.13.0+

Cause

This article provides a general guideline about restoring RDS DB instance, which is associated with a RDS service instance created through Tanzu Cloud Service broker. And the RDS DB instance has enabled the feature to have admin password being generated and managed by AWS Secrets Manager.

Resolution

In general, the end user can follow the steps below to restore the RDS DB instance from a DB snapshot when the admin password of the original DB instance is generated and managed by AWS Secrets Manager.

1. Confirm and validate a DB snapshot is available for restoration
2. Delete the original DB instance through AWS Console or CLI
3. Restore the snapshot, making sure to match the original DB instance identifier, instance class, etc.
4. Check that the restored DB instance does not have the AWS Secrets Manager integration enabled. As of this writing, from what's seen in the AWS console and CLI, user can't enable this feature during a snapshot restore. However, AWS is under constant improvement, so this may change in the future
5. Run cf update-service <RDS service instance name> --wait -c '{"use_managed_admin_password": false}' to update the CSB state to match the DB instance state
6. Run cf update-service <RDS service instance name> -c '{"use_managed_admin_password":true,"rotate_admin_password_after":7}' --wait to enable AWS Secrets Manager integration. It will fail as expected as described in the document
7. Run cf update-service <RDS service instance name> -c '{"use_managed_admin_password":true,"rotate_admin_password_after":7}' --wait again, and it will succeed

NOTE: the number "7" for property rotate_admin_password_after in above commands could be set to other value based on own need

It's not ideal to have to disable the Secrets Manager integration and then enable it again. That's something that could change as AWS make continual improvements, but the CSB product team don't have any insight into whether that's on AWS roadmap.

After the above steps are performed, a new admin password will be generated and could be retrieved from the Secrets Manager on AWS. 

The existing RDS service instance binding created through CSB before the backup should continue to work after the restore. However if this doesn't work for some reason (for example a binding created after the snapshot), then the user could just unbind the service from application and create new binding.