SCS service instance operation fails with "No X509TrustManager implementation available"

Products

• Support Only for Spring

Issue/Introduction

Sometimes SCS service instance operation (e.g. bind, unbind) might fail with error like the following.

13:44:36  An unbind operation for the service binding between app card-service and service instance my-service-registry failed: Service broker error: No X509TrustManager implementation available; nested exception is javax.net.ssl.SSLHandshakeException: No X509TrustManager implementation available

And same error message could be found in scs-service-broker job logs

2023-03-15 19:02:31.148  INFO 26 --- [           main] i.p.s.c.s.ServiceBrokerApplication       : Starting ServiceBrokerApplication using Java 1.8.0_322 on d4a74dc7-d017-45f6-9260-641d9f545ca2 with PID 26 (/var/vcap/data/p
ackages/scs-service-broker/a7e6ea9c6beb9da1f7966c7d30c36849db43a27e/scs-service-broker.jar started by vcap in /var/vcap/jobs/scs-service-broker)
2023-03-15 19:02:31.155  INFO 26 --- [           main] i.p.s.c.s.ServiceBrokerApplication       : No active profile set, falling back to default profiles: default
2023-03-15 19:02:38.100  INFO 26 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JDBC repositories in DEFAULT mode.
2023-03-15 19:02:39.264  INFO 26 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 1156 ms. Found 2 JDBC repository interfaces.
2023-03-15 19:02:55.208  INFO 26 --- [           main] i.n.util.internal.NativeLibraryLoader    : /tmp/libnetty_transport_native_epoll_x86_644155698858857993005.so exists but cannot be executed even when execute permission
s set; check volume for "noexec" flag; use -Dio.netty.native.workdir=[path] to set native working directory separately.
2023-03-15 19:02:57.194  WARN 26 --- [ctor-http-nio-2] r.netty.http.client.HttpClientConnect    : [550973e8, L:/x.x.x.x:56726 - R:api.dev.sys.yyy.com/x.x.x.x:443] The connection observed an error
javax.net.ssl.SSLHandshakeException: No X509TrustManager implementation available

The error "No X509TrustManager implementation available" indicates some kind of certificate validation failure which is caused by not finding a CA certificate to verify remote server certificate. 

During the process of deploying SCS tile, a truststore will be built to include all needed CA certificate and later scs-service-broker job would load that truststore during startup. In this case it seems that scs-service-broker hasn't loaded a complete truststore.

It's also found the error would typically happen after BBR backup is done, which seems related to this change added in SCS 3.1.32 .

• Fixed the truststore permissions issue in the mirror-service post-backup by recreating the truststore.

Eventually the problem is identified by engineering  as scs-service-broker might be started in the middle of recreating truststore during while BBR backup period.

Environment

Product Version: 3.1

Resolution

R&D is still working on the solution to ensure scs-service-broker process won't be started during BBR backup. Before the fix is ready, user could temporarily address the issue by restarting scs-service-broker job. 

1) "bosh ssh" into SCS broker instance

2) Run "sudo monit restart scs-service-broker"