Deploy-Scheduler Errand Fails - remote error: tls: unknown certificate authority

Products

• VMware Tanzu Platform - Cloud Foundry

Issue/Introduction

Deploy-Scheduler Errand failing when application ‘scheduler-broker-new’ tries to bind to MySQL service instance.

 

In the broker.stdout.logs, you'll see the certificate error below when the broker is trying to create the binding. The broker tries to first query the mysql service instance (mysql/61e5ca42-####-####-####-c400088137f5) at https://##.##.##.##:8443/status, but the request fails. This means the broker does not trust the CA that the service instance is presenting.

 

[dedicated-mysql-adapter] 2025/09/17 02:04:33 Couldn't get read only information about URL [https://##.##.##.##:8443](https://##.##.##.##:8443), error: GET [https://##.##.##.##:8443/status](https://##.##.##.##:8443/status) failed: Get "[https://##.##.##.##:8443/status](https://##.##.##.##:8443/status)": remote error: tls: unknown certificate authority

SSH onto the broker VM and retrieve the full broker.yml file

bosh -d pivotal-mysql-0643c0808f9dd0f2bee1 ssh dedicated-mysql-broker/2cc8953e-####-####-####-d5393acd8133

sudo su -

cat /var/vcap/jobs/broker/config/broker.yml

Then, ssh into the mysql service instance and run the following command to retrieve the full certificate chain.

bosh -d service-instance_0d03f617-####-####-####-81df69ea52c1 ssh mysql/ecfef145-####-####-####-bda0f5d35a36

sudo su -

openssl s_client -connect localhost:8443 -showcerts </dev/null

Compare the certs under the trusted certificates field from the broker.yml and the certificate chain on the service instance to see if they match. 

Cause

The Services TLS CA certificate is missing from the Director trusted certificates field.

As part of the services_tls_ca rotation process, the new Services TLS CA needs to be added to the Trusted Certificates field of the BOSH Director. This step may have been missed during the rotation. 

Resolution

Certificate Installation Process

1. Retrieve the services_tls_ca certificate from CredHub:

 https://techdocs.broadcom.com/us/en/vmware-tanzu/data-solutions/tanzu-for-mysql-on-cloud-foundry/3-3/mysql-for-tpcf/prepare-tls.html#add-ca-cert

credhub get -n /services/tls_ca

2. Add the certificate to the Director Trusted Certificates field

3. Redeploy all products with “Enable Upgrade all service instances” checked

https://techdocs.broadcom.com/us/en/vmware-tanzu/platform/tanzu-operations-manager/3-0/tanzu-ops-manager/security-pcf-infrastructure-advanced-certificate-rotation.html#services-rotation