Opsman (v2.10.20) upgrade failures due to failure in Credhub (v.2.9.5) database table renaming issue

Products

• Operations Manager

Issue/Introduction

OpsMan upgrade to v2.10.20, which has Credhub v.2.9.5, fails due to Credhub table renaming failure with error messages in credhub.log as follows for example:

2021-11-16T13:07:07.881Z [main] .... FATAL --- FlywayMigrationStrategyConfiguration: Error renaming migration table.
...
2021-11-16T13:07:07.923Z [main] .... ERROR --- SpringApplication: Application run failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flywayInitializer' defined in class path resource [org/springframework/boot/autoconfigure/flyway/FlywayAutoConfiguration$FlywayConfiguration.class]: Invocation of init method failed; nested exception is org.flywaydb.core.api.FlywayException: Error renaming migration table
...
Caused by: org.postgresql.util.PSQLException: ERROR: relation "flyway_schema_history" already exists

The likely cause of this error is because the customer had installed a version of Credhub with flyway6 but without the flyway5-to-flyway6 upgrade implementation that renames the table from schema_version to flyway_schmea_history in the past. When this happened, Creduhub started using the new flyway_schmea_history table ignoring the old schema_version table, which is flyway6's behavior when both tables exist. To confirm this, view the records of both schema_version table and flyway_schmea_history table in the Credhub database as follows:

Retrieve records of both flyway tables from Bosh's Credhub database:

select * from schema_version;
select * from flyway_schema_history;

Both tables should contain the schema history records. See the script column in the resultsets from both tables to make sure that flyway_schema_history has all the script records in schema_version. (The flyway_schema_history table is likely to have additional records for later versions of scripts, too). Once this is confirmed, you can go ahead to apply the remediation steps provided in this article.

 

Environment

Product Version: 2.10

Resolution

At this point, you made sure that the latest flyway schema version records were stored in the flyway_schmea_history table, which was expected because flyway6 and later ignores the old schema_version table if the new flyway_schmea_history table exists. So the proper resolution now is moving the old schema_version table out of the way as follows:

• Login to Bosh director instance and connect to credhub DB

     $/var/vcap/packages/postgres-10/bin/psql -h 127.0.0.1 -U postgres credhub

• Rename the 'schema_version' table 'backup_schema_version'

     psql> ALTER TABLE schema_version RENAME TO backup_schema_version;

• Finally, Restart the credhub server

     monit restart credhub