Credhub vm fails to start after upgrading TAS v6 with error: “SchemaManagementException: Schema-validation: missing column”

Products

• VMware Tanzu Platform - Cloud Foundry
• VMware Tanzu Application Service

Issue/Introduction

Your credhub vms are failing, verified by running command 'bosh vms' and seeing one or more of the Credhub vm's are showing 'failing/unresponsive' status. 

This issue generally happens during a second/subsequent Apply Changes after the failed first Apply Change failed on the bootstrap Credhub VM and fixed for different issue (e.g. KB 410519).  On said second/subsequent Apply Changes, the first touched credhub vm fails with similar error as below:

Caused by: jakarta.persistence.PersistenceException: [PersistenceUnit: default] Unable to build Hibernate SessionFactory; nested exception is org.hibernate.tool.schema.spi.SchemaManagementException: Schema-validation: missing column 

 

Cause

As mentioned above, the bootstrap credhub vm is left in incomplete state due to not being able to complete its original migration from said first failed Apply Changes.  Due to this 'bootstrap' vm being the first credhub vm that was touched during first Apply Changes and the Flyway migrations did not complete, the second/subsequent Apply Changes bosh starts with the next (non-bootstrap) Credhub vm. Because Flyway is diabled on said non-bootstrap vm:

1. The VM does not validate or migrate the schema.
2. Since the schema is still incomplete/inconsistent (ie. the missing coulmns) it fails with: SchemaManagementException: Schema-validation: missing column

Note: *This issue can be confirmed by checking the failed Credhub vm logs, in which there will be no evidence of a Flyway migration being ran. See example below of the failed Credhub VM logs:

INFO --- DatatabaseLayerImpl: Checking for existence of 'schema_version' table: false
[main] ....  INFO --- FlywayMigrationStrategyConfiguration: Validating database state...
[main] ....  INFO --- FlywayExecutor: Database: jdbc:mariadb://mysql.service.cf.internal:3306/credhub
?autoReconnect=true&socketTimeout=3600000 (MySQL 8.0)
[main] ....  INFO --- DbValidate: Successfully validated 74 migrations (execution time 00:00.129s)
[main] ....  INFO --- FlywayMigrationStrategyConfiguration: Validation successful.
[main] ....  INFO --- FlywayMigrationStrategyConfiguration: Running FlyWay migration....
[main] ....  INFO --- DbValidate: Successfully validated 74 migrations (execution time 00:00.064s)

 

Related Context:

Credhub utilizes Flyway tool for its database schema validation during vm startup. Usually in a TAS deployment, the Credhub vm marked as sequence number "0" is configured to run Flyway migrations.  Further verify by checking the following config in the vm directory: /var/vcap/jobs/credhub/config/application/spring.yml as the example shows below:

spring:
  flyway:
    enabled: false

Resolution

Complete the below steps (in order):

1. Identify if credhub bootstrap node is detached: bosh -d <CF_DEPLOYMENT> instances --details --column={Instance,State,Bootstrap} | grep -e 'credhub.*detached.*true'
   • Note: If that command outputs/shows a credhub instance as (State:"detached"), follow the KB to fix detached instance: https://knowledge.broadcom.com/external/article/293796/
2. Recreate the bootstrap vm: bosh recreate <vm> --no-converge
3. Re-run Apply Changes
   • This will allow the bootstrap vm the Flyway migration to insert the missing column and should start as desired.
4. After this Apply Changes, confirm all credhub vms are in 'running' state: bosh is -I | grep credhub