Config Server stops being able to connect to credhub: readHandshakeRecord

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

It may happen that an app bound to a Config Server can't read from it anymore, and Config Server showing errors containing "readHandshakeRecord" when connecting to Credhub. E.g

"2023-06-12T04:00:47.539+0000","[35m[http-nio-8080-exec-9][0;39m [1;31mERROR[0;39m o.a.c.c.C.[.[.[.[dispatcherServlet].log - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.web.client.ResourceAccessException: I/O error on GET request for ""https://credhub.service.cf.internal:8844/api/v1/data"": readHandshakeRecord; nested exception is javax.net.ssl.SSLException: readHandshakeRecord] with root cause" "2023-06-12T04:00:21.242+0000","[35m[http-nio-8080-exec-8][0;39m [1;31mERROR[0;39m o.s.b.a.a.c.s.CloudFoundrySecurityInterceptor.preHandle - org.springframework.boot.actuate.autoconfigure.cloudfoundry.CloudFoundryAuthorizationException: Access denied" "2023-06-12T03:58:29.553+0000","[35m[http-nio-8080-exec-4][0;39m [1;31mERROR[0;39m o.s.b.a.a.c.s.CloudFoundrySecurityInterceptor.preHandle - org.springframework.boot.actuate.autoconfigure.cloudfoundry.CloudFoundryAuthorizationException: Access denied" "2023-06-12T03:57:12.036+0000"," at org.cloudfoundry.router.ClientCertificateMapper.doFilter(ClientCertificateMapper.java:79)" "2023-06-12T03:57:12.036+0000"," at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)" "2023-06-12T03:57:12.036+0000"," at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)"

 There is a known issue with certs in Java builpack with the Container Security Provider library, in which a race condition could result in mismatched private-key and certificate pairs when Diego rotated these credentials for the container that can lead to this issue. See this issue for more details.

This is fixed in Java builpack  v4.57 .

Resolution

For an immediate solution, restart the config server and the application. The re-occurrence of this issue is very low, so after restarting, it may not happen again in a long time, if ever. 

For a permanent fix, update Java builpack to v4.57 or above.