Push-Usage service errand fails with certificate error in TAS 2.9.x when using external database AWS RDS / GCP

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

***This issue is fixed in versions 2.9.4, 2.10 and 2.11**

When running errand push-usage-service, the errand fails to connect to external data-base AWS/RDS and fails with a certificate error similar to the following output: 

2020-08-20T18:38:04.47+0000 [APP/TASK/48ee9bdd/0] ERR rake aborted!
2020-08-20T18:38:04.47+0000 [APP/TASK/48ee9bdd/0] ERR Mysql2::Error: SSL connection error: unable to get local issuer certificate
2020-08-20T18:38:04.47+0000 [APP/TASK/48ee9bdd/0] ERR /home/vcap/deps/0/vendor_bundle/ruby/2.6.0/gems/mysql2-0.4.10/lib/mysql2/client.rb:89:in `connect'
2020-08-20T18:38:04.47+0000 [APP/TASK/48ee9bdd/0] ERR /home/vcap/deps/0/vendor_bundle/ruby/2.6.0/gems/mysql2-0.4.10/lib/mysql2/client.rb:89:in `initialize'
2020-08-20T18:38:04.47+0000 [APP/TASK/48ee9bdd/0] ERR /home/vcap/deps/0/vendor_bundle/ruby/2.6.0/gems/activerecord-5.2.4.3/lib/active_record/connection_adapters/mysql2_adapter.rb:22:in `new'
2020-08-20T18:38:04.47+0000 [APP/TASK/48ee9bdd/0] ERR /home/vcap/deps/0/vendor_bundle/ruby/2.6.0/gems/activerecord-5.2.4.3/lib/active_record/connection_adapters/mysql2_adapter.rb:22:in `mysql2_connection'

 

Environment

Product Version: 2.9

Resolution

There were changes introduced recently to make sure the service respected the properties.system_database.external.validate_hostname.value in p-runtime, but in doing so it seems there were some un-intended consequences such as this issue here.

There are 2 ways in which you can resolve this issue:

1. Uncheck "Enable hostname validation" in the PAS tile under "Databases" and apply changes to the PAS tile.
 

2. For AWS RDS database you can add the AWS/RDS Root CA bundle to the Bosh -> Security section of the BOSH tile.

The AWS/RDS Root CA bundle can be found here: AWS SSL/TLS  
 

1. Download the CombinedCABundle pem file which includes the AWS RDS Root CA and all associated leaf certs.

2. Add the contents of the pem file to the Security tab in the BOSH Director tile
 

3. Apply Changes to BOSH Director and TAS tile. Once BOSH has updated the clock_global VMs with the new certificates, the push-usage-service errand should be able to connect to the RDS database.