Issues upgrading Tanzu Bosh Director with errors: "Waiting for instance 'bosh/0' to be running... Failed"

Products

• Operations Manager
• VMware Tanzu Application Service

Issue/Introduction

• Bosh upgrade fails during Apply Changes operation, Ops Manager shows failures like:

  Updating instance 'bosh/0'... Finished (00:01:18)

  Waiting for instance 'bosh/0' to be running... Failed (00:05:01)

Failed deploying (00:09:38)

 

Exit code 1

===== 2025-05-08 16:22:42 UTC Finished "/usr/local/bin/bosh --no-color --non-interactive --tty create-env /var/tempest/workspaces/default/deployments/bosh.yml -l /var/tempest/workspaces/default/deployments/vars.yml --package-dir /var/vcap/packages"; Duration: 602s; Exit Status: 1

 

• The support bundle Bosh Director logging will show errors in the bosh-nats-sync.log:

E, [2025-05-18T11:45:37.359937 #13309] ERROR : Failed to obtain token from UAA: #<CF::UAA::BadTarget: error: Connection refused - Connection refused - connect(2) for "<BOSH_IP>" port 8443 (<BOSH_IP>:8443)>

 

 

• The support bundle Bosh Director logging will show errors in the uaa.log:

[2025-05-19T15:38:40.094029Z] uaa - 12 [main] - [,] .... ERROR --- DbMigrate: Migration of schema "public" to version "4.110 - Add ExternalKey To Identity Provider" failed! Changes successfully rolled back.

 

Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.flywaydb.core.Flyway]: Factory method 'flyway' threw exception; nested exception is org.flywaydb.core.internal.command.DbMigrate$FlywayMigrateException: Migration V4_110__Add_ExternalKey_To_Identity_Provider.sql failed

-----------------------------------------------------------------

SQL State  : 42701

Error Code : 0

Message    : ERROR: column "external_key" of relation "identity_provider" already exists

Location   : org/cloudfoundry/identity/uaa/db/postgresql/V4_110__Add_ExternalKey_To_Identity_Provider.sql (/var/vcap/jobs/uaa/file:/var/vcap/data/uaa/tomcat/webapps/ROOT/WEB-INF/lib/cloudfoundry-id

 

 

Environment

This error was observed when upgrading Bosh Director 3.0.30-build.1377 to Bosh Director 3.0.40-build.1808. It is not limited to these versions.

Cause

During certain upgrade versions on Bosh Director, the UAA database schema is updated. When the schema update script runs, an entry is added to the schema_versions table indicating any DELETE operations that are performed. If the upgrade is interrupted for any reason, these entries are retained in the database schema_versions table, which prevents the upgrade from progressing successfully.

Resolution

Corrective action requires backing up the UAA database from Bosh SSH (preferably using BBR backup), then manually deleting the offending entries from the schema_version table in the UAA database. This is a high risk operation, please contact Broadcom support for assistance.