Credhub fails to connect to remote RDS MySQL database over TLS

Products

• Operations Manager

Issue/Introduction

When upgrading TAS tile to version 2.8.x credhub instance is required as mentioned in release note. However sometimes updating credhub instance might fail as shown below.

Task 114382 | 20:02:34 | Updating instance credhub: credhub/9d5c6183-7449-463c-9e7d-d28c9a4e8f41 (0) (canary) (00:05:20) 
L Error: 'credhub/9d5c6183-7449-463c-9e7d-d28c9a4e8f41 (0)' is not running after update. Review logs for failed jobs: credhub 

And error "Unsupported record version Unknown-0.0" was seen in credhub logs.

SQL State  : 08
Error Code : -1
Message    : Could not connect to example.rds.amazonaws.com:3306 : Unsupported record version Unknown-0.0

        at org.flywaydb.core.internal.jdbc.JdbcUtils.openConnection(JdbcUtils.java:60) ~[flyway-core-5.2.4.jar!/:?]
        at org.flywaydb.core.internal.database.DatabaseFactory.createDatabase(DatabaseFactory.java:72) ~[flyway-core-5.2.4.jar!/:?]
        at org.flywaydb.core.Flyway.execute(Flyway.java:1670) ~[flyway-core-5.2.4.jar!/:?]
        at org.flywaydb.core.Flyway.migrate(Flyway.java:1356) ~[flyway-core-5.2.4.jar!/:?]
        at org.springframework.boot.autoconfigure.flyway.FlywayMigrationInitializer.afterPropertiesSet(FlywayMigrationInitializer.java:65) ~[spring-boot-autoc
onfigure-2.2.1.RELEASE.jar!/:2.2.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1862) ~[spri
ng-beans-5.2.1.RELEASE.jar!/:5.2.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1799) ~[spring-
beans-5.2.1.RELEASE.jar!/:5.2.1.RELEASE]
......

Environment

Product Version: 2.8

Resolution

For security reason credhub communicates with database over TLS and it requires external database to support TLS v1.2. However some old version of RDS MySQL database instance doesn't support TLS v1.2. That's why credhub complained "Unsupported record version Unknown-0.0" when connecting to RDS database. 

The solution is to check RDS MySQL database and upgrade it if the current version doesn't support TLS v1.2.

Another thing needs to be checked is "Database CA certificate" configured on Credhub pane of TAS setting page. If a certificate bundle is downloaded from AWS site and multiple certificates are specified, then "unable to find valid certification path to requested target" error might be seen since only the first certificate in bundle will be imported into truststore. More discussions are available here.
 
So it's suggested to use the single certificate from AWS to put into "Database CA certificate" box.