Scheduler deployment is failing at errand with exception/error 'java.security.cert.CertPathValidatorException: signature check failed'

Products

• Operations Manager
• VMware vSphere 7.0 with Tanzu

Issue/Introduction

The below error occurs on Apply Change log:

'For application 'scheduler-broker-new': For service 'scheduler-broker-mysq]': Service broker error: There was a problem completing your request. Please contact your operations team providing the following information: service: p-mysql, service-instance-guid: <#GUID-ID>, broker-request-id: <REQUEST-ID>, operation:'

1 errand(s)
===== 2024-11-06 17:15:07 UTC Finished "/usr/local/bin/bosh --no-color --non-interactive --tty --environment=10.49.56.10 --deployment=p-scheduler-2c7336736d0a16d20667 run-errand deploy-scheduler"; Duration: 3725; Exit Status: 1
Exited with 1.

 

The below is an example of the error from Scheduler/broker app that fails due to the issue:

# notice the: "Unable to obtain connection from database: Communications link failure".

[org/springframework/boot/autoconfigure/orm/jpa/HibernateJpaConfiguration.class]: Failed to initialize dependency 'flywayInitializer' of LoadTimeWeaverAware bean 'entityManagerFactory': Error creating bean with name 'flywayInitializer' defined in class path resource [org/springframework/boot/autoconfigure/flyway/FlywayAutoConfiguration$FlywayConfiguration.class]: Unable to obtain connection from database: Communications link failure
2024-11-05<#TimeStamp#> [APP/PROC/WEB/1] OUT The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
java.security.cert.CertPathValidatorException: signature check failed.

 

Cause

The error can occur when a container can not connect to the desired database, and can be influenced by either of the below conditions:

1. An expired certificate on opsman VM - >Bosh tile -> Security -> trusted certs on the opsman VM user interface.
2. If there is more than one cert with the same Issuer/Name/Subject in the Bosh tile -> Security -> trusted certs on the opsman VM.  

Resolution

Check the "Trusted certs" on Opsman UI -> Bosh tile- >Security Tab or with the cli (openssl) to ensure there is only one CA cert that is current and in place.

1. If there is an expired cert then will need to update and re-apply changes.
2. If there is more than one "trusted cert" in the Opsman UI -> Bosh tile- >Security Tab, then removing the extra cert will be necessary.  Be sure to save your changes from the Opsman UI -> Bosh tile- >Security Tab, then will need to run "Apply Changes" from opsman to ensure the desired cert is removed to re-establish trust.

Additional Information

1.The cert use to access mysql db is 'services/tls_ca'
2.Verify the the current 'services/tls_ca' from credhub. This can be used to validate the trusted certs in Bosh tile -> Security -> trusted certs

credhub find -n services/tls_ca

credhub get -n KEY_NAME