Upgrading Tanzu Cloud Service Broker for AWS to version 1.13 fails with pre-start script of csb-app job
search cancel

Upgrading Tanzu Cloud Service Broker for AWS to version 1.13 fails with pre-start script of csb-app job

book

Article ID: 383161

calendar_today

Updated On:

Products

Operations Manager

Issue/Introduction

Starting from release v1.13, the Cloud Service Broker (CSB) for AWS is a BOSH-managed virtual machine and is deployed as part of the main job within the BOSH deployment lifecycle. When upgrading this CSB tile to v1.13, it might fail with pre-start script of csb-app job as shown below.

Task 1417493 | 11:27:16 | Preparing deployment: Preparing deployment (00:00:00)
Task 1417493 | 11:27:16 | Preparing deployment: Rendering templates (00:00:02)
Task 1417493 | 11:27:18 | Preparing package compilation: Finding packages to compile (00:00:00)
Task 1417493 | 11:27:19 | Updating instance cloud-service-broker: cloud-service-broker/4dd3f256-####-####-####-c90466f46417 (0) (canary)
Task 1417493 | 11:27:32 | L executing pre-stop: cloud-service-broker/4dd3f256-####-####-####-c90466f46417 (0) (canary)
Task 1417493 | 11:27:32 | L executing drain: cloud-service-broker/4dd3f256-####-####-####-c90466f46417 (0) (canary)
Task 1417493 | 11:27:33 | L stopping jobs: cloud-service-broker/4dd3f256-####-####-####-c90466f46417 (0) (canary)
Task 1417493 | 11:27:34 | L executing post-stop: cloud-service-broker/4dd3f256-####-####-####-c90466f46417 (0) (canary)
Task 1417493 | 11:27:36 | L installing packages: cloud-service-broker/4dd3f256-####-####-####-c90466f46417 (0) (canary)
Task 1417493 | 11:27:39 | L configuring jobs: cloud-service-broker/4dd3f256-####-####-####-c90466f46417 (0) (canary)
Task 1417493 | 11:27:39 | L executing pre-start: cloud-service-broker/4dd3f256-####-####-####-c90466f46417 (0) (canary) (00:00:22)
Updating deployment:
 Expected task '1417493' to succeed but state is 'error'
Exit code 1
                       L Error: Action Failed get_task: Task 67e47a3e-####-####-####-ec0572b8d6b4 result: 1 of 4 pre-start scripts failed. Failed Jobs: csb-app. Successful Jobs: falcon-linux-sensor, bpm, bosh-dns.
Task 1417493 | 11:27:41 | Error: Action Failed get_task: Task 67e47a3e-####-####-####-ec0572b8d6b4 result: 1 of 4 pre-start scripts failed. Failed Jobs: csb-app. Successful Jobs: falcon-linux-sensor, bpm, bosh-dns.

csb-app pre-start script logs show the pre-install check failure for some service instances. For example,

Plugin UpgradeAllServices N/A successfully installed.
2024-12-02T00:08:05Z: discovering service instances for broker: cloud-service-broker-aws
2024-12-02T00:08:05Z: skipping instance: "csb1" guid: "0ea8fd21-####-####-####-16cc9d85b212" Upgrade Available: true Last Operation Type: "create" State: "failed"
2024-12-02T00:08:05Z: skipping instance: "csb2" guid: "3354d8af-####-####-####-7349dafb922b" Deactivated Plan: "small" Offering: "csb-aws-mssql" Offering guid: "04c0557b-####-####-####-5c3eec97605d" Upgrade Available: false Last Operation Type: "create" State: "failed"
......
Pre-install check failed. There are service instances pending upgrade to a previous version. Please ensure all service instances are up to date before retrying the installation. Read the documentation for detailed instructions in the upgrading section of the previous release needed to upgrade: https://docs.vmware.com/en/Cloud-Service-Broker-for-VMware-Tanzu/index.html.

 

Environment

Tanzu Cloud Service Broker for AWS v1.13

Cause

According to the release note, there are some breaking changes in release v1.13.

Breaking Changes

  • Deploy-all errand no longer exists: The deploy-all errand has been removed from Cloud Service Broker for AWS. Previously, this errand was used to deploy the broker. Now, the broker is a BOSH-managed virtual machine and is deployed as part of the main job within the BOSH deployment lifecycle.

  • You cannot upgrade the service broker if service instances are not up-to-date: A step in the installation phase stops the process if there are any outdated instances.

    The Cloud Service Broker will not start if there are any instances that are not upgraded. Therefore, ensure that all instances are upgraded before staging the new version of the tile.

    For these reasons, verify that there are no service instances pending upgrade before staging the new version of the tile.

    You can see whether there are instances with a pending upgrade by using the CLI plug-in. For more information, see Verify that all instances are up to date with the CLI plug-in.

The pre-start script of csb-app job will run the following commands one by one to check if all existing service instances are upgradable. The script will fail if the second command (-min-version-required) returns non-success. 

cf upgrade-all-services cloud-service-broker-aws -check-deactivated-plans
cf upgrade-all-services cloud-service-broker-aws -min-version-required

Resolution

Check the current status of CSB service instances reported by pre-start script. If they are already orphaned instances, try to delete or purge them by referring to this document. 

Powered by Wolken Software