Cloud Service Broker Upgrade All Services Errand fails with DBInstanceAlreadyExist

Products

• VMware Tanzu Application Service

Issue/Introduction

You might have upgraded Cloud Service Broker (CSB)  to a more recent version or you might have changed tile settings. You "Apply Change" with the Upgrade Service Instance Errand turned on and if fails with DBInstanceAlreadyExists error

A sample log below

Details: "upgrade failed: Error: updating RDS DB Instance (csb-postgresql-1234-####): operation error RDS: ModifyDBInstance, https response error StatusCode: 400, RequestID: b70a28e0-####-####-####, DBInstanceAlreadyExists: There is already an DB Instance named ascsb-postgresql-1234-####. Please choose another name. with aws_db_instance.db_instance, on main.tf line 51, in resource \"aws_db_instance\" \"db_instance\": 51: resource \"aws_db_instance\" \"db_instance\" { exit status 1"

Cause

Check other postgress instances:

aws rds describe-db-instances --filters Name=engine,Values=postgres

 

Look for csb-postgresql-1234-####

{
            "DBInstanceIdentifier": "csb-postgresql-1234-####",
            "DBInstanceClass": "db.t4g.medium",
             ...

             "TagList": [
             {
                    "Key": "pcf-instance-id",
                    "Value": "ABCD-####"
                },

...

},

Take note of the pcf-instance-id value, you are like to get another db with the same pcf-instance-id

{            
            "DBInstanceIdentifier": "csb-postgresql-1234-45678-####",
            "DBInstanceClass": "db.t4g.medium",
             ...
            "TagList": [
            {
                    "Key": "pcf-instance-id",
                    "Value": "ABCD-####"
                },

So when you try to upgrade an instance it fails with DBInstanceAlreadyExists

Resolution

The error suggests that there was an experiment to backup and restore this service instance. The service instance has been backed up, deleted, and restored - but there's a small discrepancy in the restore that means the OpenTofu provider has not identified it as the existing instance, and is instead trying to re-create it, but with the pcf-instance-id, the service instance would not have been usable prior to the upgrade.

 

1. If the database csb-postgresql-1234-45678-#### is not being used, You may delete csb-postgresql-1234-45678-####.

2. Once deleted, you can Apply change with the Upgrade service instance errand. This will now succeed as there will only be one database with pcf-instance-id "ABCD-####"

3. If the database csb-postgresql-1234-45678-#### cannot be deleted, You can use the following command to test restoring snapshot to get more info

aws rds restore-db-instance-from-db-snapshot --db-snapshot-identifier snapshot-restore-test-stitch-dancer --db-instance-identifier csb-postgresql-1234-#### --db-instance-class db.t4g.medium

Please refer to AWS documentation or reach out to AWS Support for guidance/troubleshooting guide on restoring db.