Orphaned services appear in Cloud Service Broker IaaS after upgrading CSB tile to v1.6.0

Products

• VMware Tanzu Application Service

Issue/Introduction

1.) A user completes all the 'Pre-Upgrade Procedures/Checks' from the official docs, then successfully upgrades 'Tanzu Cloud Service Broker for GCP tile' to v1.6.0.   

# Prior to upgrade:

cf upgrade-all-services cloud-service-broker-gcp -check-up-to-date
<TIMESTAMP>: discovering service instances for broker: cloud-service-broker-gcp
<TIMESTAMP>: ---
<TIMESTAMP>:  total instances: 89
<TIMESTAMP>:  upgradable instances: 0
<TIMESTAMP>: ---
<TIMESTAMP>: starting upgrade...
<TIMESTAMP>: upgraded 0 of 0
<TIMESTAMP>:  ---
<TIMESTAMP>:  skipped 0 instances
<TIMESTAMP>:  successfully upgraded 0 instances

 

2.) After successfully upgrading the tile via Apply Changes from opsman, the user notices that some services instances appear in their IaaS UI (GCP) that were not there prior.

 

3.) Running the same pre-upgrade command (cf upgrade-all-services cloud-service-broker-gcp -check-up-to-date) post upgrade will confirm the issue.

# After upgrade:

<TIMESTAMP>: cf upgrade-all-services cloud-service-broker-gcp -check-up-to-date
<TIMESTAMP> discovering service instances for broker: cloud-service-broker-gcp
<TIMESTAMP> skipping instance: "<EXAMPLE-uat>" guid: "e5###!!!-b##$-4aae-###7-3e###$$$!!!" Upgrade Available: true Last Operation: Type: "create" State: "failed"
<TIMESTAMP> skipping instance: "<EXAMPLE-qa>" guid: "a###$$$!-8$$#-4!#$-8$$$-c!!!!!$$$$#4" Upgrade Available: true Last Operation: Type: "create" State: "failed"
<TIMESTAMP> no instances available to upgrade

# Notice how "skipping instance" did not appear in the output of command 'cf upgrade-all-services cloud-service-broker-gcp' during the Pre-Upgrade procedures/prior to the upgrade. Also take note of the status of the 'skipped instances' output: "Upgrade Available: true Last Operation: Type: "create" State: "failed; no instances available to upgrade".  This indicates/means the creation was initiated with CSB version prior to upgrade.

Cause

This is the expected behavior. When attempting to upgrade these instances, the broker explicitly skipped them because they were in a failed creation state, meaning there was nothing to upgrade. As a result of the service instances being in a failed creation state prior to the upgrade, they appear on the IaaS UI post upgrade.

Resolution

Workaround:

Since these instances are marked as failed in Cloud Foundry but still appear in the GCP console, they are considered orphaned. The fix is to Remove the resources/service instances in GCP manually via the Google Cloud Console, and then running the below command to remove them from Cloud Foundry:

`cf delete-service <instance-name>` 
# or 
`cf purge-service-instance <instance-name>` 

 

Resolution: 

To resolve the known issue (called out in release notes of CSB for GCP v1.7.0), upgrade to CSB v1.7.x. 

Additional Information

If there are service instance references that appear while performing the Pre-Upgrade checks (prior to upgrade), then please see the documentation to manually upgrade the instances before proceeding with the tile upgrade/Apply Changes. If this is not successful, then is recommended to open a ticket with TANZU Support for assistance.