Cloud Controller Pre-Start Fails During TAS v4.0.x to v6.0.x Upgrade

Products

• VMware Tanzu Application Service

Issue/Introduction

During a TAS upgrade from v4.0.x to v6.0.x, it may fail on the cloud_controller pre-start:

Task 10565 | 12:53:13 | Updating instance cloud_controller: cloud_controller/<REDACTED> (0) (canary)
Task 10565 | 12:53:13 | L executing pre-stop: cloud_controller/<REDACTED> (0) (canary)
Task 10565 | 12:53:13 | L executing drain: cloud_controller/<REDACTED> (0) (canary)
Task 10565 | 12:53:21 | L stopping jobs: cloud_controller/<REDACTED> (0) (canary)
Task 10565 | 12:53:23 | L executing post-stop: cloud_controller/<REDACTED> (0) (canary)
Task 10565 | 12:56:13 | L installing packages: cloud_controller/<REDACTED> (0) (canary)
Task 10565 | 13:08:10 | L configuring jobs: cloud_controller/<REDACTED> (0) (canary)
Task 10565 | 13:08:10 | L executing pre-start: cloud_controller/<REDACTED> (0) (canary) (00:10:43)
                        L Error: Action Failed get_task: Task <REDACTED> result: 1 of 8 pre-start scripts failed. Failed Jobs: cloud_controller_ng. Successful Jobs: loggregator_agent, bpm, bosh-dns, syslog_forwarder, antivirus.
Task 10565 | 13:10:03 | Error: Action Failed get_task: Task <REDACTED> result: 1 of 8 pre-start scripts failed. Failed Jobs: cloud_controller_ng. Successful Jobs: loggregator_agent, bpm, bosh-dns, syslog_forwarder, antivirus.

 

On the cloud_controller instance that failed, /var/vcap/sys/log/cloud_controller_ng/pre-start.stderr.log shows errors running the database migrations

[2024-12-03 13:08:40+0000] + echo 'Running migration try number 3 of 3'
[2024-12-03 13:08:40+0000] + bundle exec rake db:migrate
[2024-12-03 13:08:45+0000] rake aborted!
[2024-12-03 13:08:45+0000] Sequel::Migrator::Error: Applied migration files not in file system: 20240820070742_add_jobs_user_guid_state_index.rb (Sequel::Migrator::Error)

 

Cause

The error "Applied migration files not in file system" indicates that cloud controller may have been downgraded and doesn't know about this migration. This can happen when upgrading from certain TAS v4.0.x versions to certain v6.0.x versions.

For example, if upgrading from TAS v4.0.28 to v6.0.7, CAPI will be downgraded from v1.193.0 to v1.189.0 and you will run into migration errors.

Resolution

Check the TAS component versions in the release notes and upgrade to a version that contains the same or higher version of CAPI. You should verify the other components are the same or higher version too.

For example, if upgrading from TAS v4.0.28, you should upgrade to v6.0.9 or later.

TAS v4.0 release notes: https://techdocs.broadcom.com/us/en/vmware-tanzu/platform/tanzu-platform-for-cloud-foundry/4-0/tpcf/runtime-rn.html

TAS v6.0 release notes: https://techdocs.broadcom.com/us/en/vmware-tanzu/platform/tanzu-platform-for-cloud-foundry/6-0/tpcf/runtime-rn.html