Applying changes are failing when upgrading TAS from 2.11.37 to 2.12.24

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

When upgrading TAS from 2.11.37 to 2.12.24 apply changes keep failing and bbs process is failing at all diego_database instances. The installation logs for the failure are like the following.

Task 2819568 | 14:49:02 | Preparing deployment: Preparing deployment (00:00:32)
Task 2819568 | 14:49:34 | Preparing deployment: Rendering templates (00:03:53)
Task 2819568 | 14:53:31 | Preparing package compilation: Finding packages to compile (00:00:02)
Task 2819568 | 14:53:45 | Updating instance diego_database: diego_database/be0f0eb4-dff0-4834-b5e5-17b2d47cc1ee (0) (canary)
Task 2819568 | 14:53:45 | L executing pre-stop: diego_database/be0f0eb4-dff0-4834-b5e5-17b2d47cc1ee (0) (canary)
Task 2819568 | 14:53:46 | L executing drain: diego_database/be0f0eb4-dff0-4834-b5e5-17b2d47cc1ee (0) (canary)
Task 2819568 | 14:54:00 | L stopping jobs: diego_database/be0f0eb4-dff0-4834-b5e5-17b2d47cc1ee (0) (canary)
Task 2819568 | 14:54:26 | L executing post-stop: diego_database/be0f0eb4-dff0-4834-b5e5-17b2d47cc1ee (0) (canary)
Task 2819568 | 14:56:35 | L installing packages: diego_database/be0f0eb4-dff0-4834-b5e5-17b2d47cc1ee (0) (canary)
Task 2819568 | 14:57:01 | L configuring jobs: diego_database/be0f0eb4-dff0-4834-b5e5-17b2d47cc1ee (0) (canary)
Task 2819568 | 14:57:01 | L executing pre-start: diego_database/be0f0eb4-dff0-4834-b5e5-17b2d47cc1ee (0) (canary)
Task 2819568 | 14:57:40 | L starting jobs: diego_database/be0f0eb4-dff0-4834-b5e5-17b2d47cc1ee (0) (canary) (00:08:57)
                        L Error: 'diego_database/be0f0eb4-dff0-4834-b5e5-17b2d47cc1ee (0)' is not running after update. Review logs for failed jobs: bbs
Task 2819568 | 15:02:42 | Error: 'diego_database/be0f0eb4-dff0-4834-b5e5-17b2d47cc1ee (0)' is not running after update. Review logs for failed jobs: bbs

When reviewing bbs job logs on the diego_database instance could see error indicating the existing DB version is higher that will be upgraded to.

{"timestamp":"2023-05-19T14:58:03.101301587Z","level":"error","source":"bbs","message":"bbs.exited-with-failure","data":{"error":"Exit trace for group:\nmigration-manager exited with error: Existing DB version (1676360874) exceeds bbs version (1643660541)\nserver exited with nil\nworkpool exited with nil\nset-lock-held-metrics exited with nil\nlock exited with nil\nlock-held-metrics exited with nil\nperiodic-filedescriptor-metrics exited with nil\nhealthcheck exited with nil\ndebug-server exited with nil\n"}}

Then checking the installation logs regarding manifest change, it's shown that Diego release is actually downgraded with the deployment. That's why the above error is produced by bbs.

  - name: diego
    exported_from:
+   - os: ubuntu-xenial
+     version: '621.418'
-   - os: ubuntu-xenial
-     version: '621.488'
-   url: file:///var/tempest/releases/diego-2.73.0-ubuntu-xenial-621.488.tgz
+   url: file:///var/tempest/releases/diego-2.71.0-ubuntu-xenial-621.418.tgz
-   version: 2.73.0
+   version: 2.71.0

This is because TAS 2.12 release is already at EOGS (End Of General Support) stage, which means it will not be patched with update-to-date component releases. However TAS 2.11 is still LTS (Long Term Support) release and will continue getting patches for all its components. For the aforementioned upgrade path TAS 2.11.37 was released on 04/20/2023 while latest TAS 2.12 release v2.12.24 was out on 02/28/2023.  Therefore most components (including Diego) in 2.11.37 are newer than that in 2.12.24. 

Environment

Product Version: 2.12

Resolution

It's recommended to use Upgrade Planner tool to plan the upgrade path. Avoid upgrading TAS from 2.11 to 2.12 and choosing 2.13 as target release instead. 

In this case TAS was already staged to 2.12.24 and not able to be reverted back to 2.11.37, it's suggested to do roll upgrade to TAS 2.13 directly.