VMware SQL with MySQL for Tanzu Application Service: Smoke test fails with tls certificate error

Products

• VMware Tanzu SQL

Issue/Introduction

When a user does Apply Change on VMware SQL with MySQL for Tanzu Application Service tile, the smoke test might fail with Service broker error as shown. Customers faced the issue after appending the new custom certificates into the Ops Manager UI - BOSH Director - Security - Trusted Certificates. 

Service broker error: There was a problem when connecting to a MySQL server. Check database availability, persistent disk usage, and network availability, then try again. 

In pivotal-mysql deployment - dedicated-mysql-broker vm - /var/vcap/sys/log/broker/broker.stdout.log, remote error: tls: bad certificate could be found.
Here is an example:

failed: Get "https://IP:PORT/status": remote error: tls: bad certificate

The IP here is the new created mysql service instance ip.

Print SSL certificate by running below openssl command:

openssl s_client -showcerts -connect IP:PORT </dev/null

The result shows the cert is not expired, but it contains below error:

Verify return code: 21 (unable to verify the first certificate)

Environment

Product Version: 2.10

Resolution

Explanation:
This issue is caused by the tls ca is not added into the trusted certificates.

Fix:

• Log in to Credhub

Please refer Accessing BOSH CredHub with the CredHub CLI 
Or 
vcap user not able to ssh into BOSH Director and How to access Ops Manager's CredHub with the CredHub CLI

• Record the CA certificate by running:

credhub get \
  --name=/services/tls_ca \
  -k ca

• Navigate to the Ops Manager Installation Dashboard > BOSH Director > Security.
• Append the contents of the CA certificate you recorded in the previous step into Trusted Certificates.
• Click Save.
• Apply Change tiles 

Please also refer: Add the CA Certificate