Java app on Tanzu Application Service raise javax.net.ssl.SSLHandshakeException when connecting to remote endpoint

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

When Java apps on Tanzu Application Service (TAS) reach out to remote endpoints over the HTTPS protocol, the connections may fail due to TLS handshake javax.net.ssl.SSLHandshakeException: <message>. 

The exception usually comes with some detailed error message such as the one below: 
 

"General SSLEngine problem ... PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"

javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1521) ~[na:1.8.0_222]
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:528) ~[na:1.8.0_222]
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:802) ~[na:1.8.0_222]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766) ~[na:1.8.0_222]
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) ~[na:1.8.0_222]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) ~[na:1.8.0_222]

"Received fatal alert: unknown_ca"

2020-10-01 17:51:49.808 ERROR 30 --- [io-8080-exec-15] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is feign.RetryableException: Received fatal alert: unknown_ca executing GET https://****** with root cause
javax.net.ssl.SSLHandshakeException: Received fatal alert: unknown_ca
at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source) ~[na:na]
at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source) ~[na:na]
at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source) ~[na:na]

"Received fatal alert: bad_certificate"

2020-06-08T15:23:53.46+0200 [APP/PROC/WEB/0] OUT Caused by: org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://*****": Received fatal alert: bad_certificate; nested exception is javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate

Resolution

The JVM and Java are different from other programming languages in that the JVM maintains its own store of trusted certificate authorities. 

An exception like javax.net.ssl.SSLHandshakeException indicates a TLS handshake failure while your Java app is connecting to remote a endpoint over HTTPS. To figure out why it failed, you need to look at the message of the exception, as each error message indicates a different cause. 

 

General SSLEngine problem ... PKIX path building failed: ***

When establishing a TLS connection, the Java app downloads the server's TLS certificate from the remote endpoint and verifies it using the set of trusted certificate authorities in the JVM's truststore. For apps on TAS, the Java buildpack will import the container's list of trusted certificate authorities into the JVM's truststore automatically (this includes Bosh Trusted Certificates).

In general, if the server's certificate is signed by a trusted public CA it'll just work. If you have a server certificate that is signed by a private or internal CA, and you are getting this error you will need to take some additional steps to make sure the certificate is trusted. Please refer to the following KB:

How to tell application containers running Java apps to trust self-signed certifications or a private or internal CA.

 

javax.net.ssl.SSLHandshakeException: Received fatal alert: ***

All Received fatal alert: errors are caused by unsuccessful mutual TLS handshakes. In the same way that the client verifies the server's certificate, the server may also try to verify the client's certificate if one is present.

For Java applications running on TAS, the Java buildpack will automatically send a client certificate with most HTTPS requests that are made. The certificate it sends is the Diego instance-id certificate, i.e. /etc/cf-instance-credentials directory from the app container, and this key pair is updated every 24 hours automatically.

If your server requests or requires a client certificate, the instance certificate will be sent out as a client certificate by the Java app. Again, this happens automatically for most HTTPS requests made by an application.

Because the instance-id certificate is signed by the Diego Instance Root CA, this CA must be deployed onto the remote server or load balancer that terminates TLS. If it is not, then the client won't be verified by the server/load balancer, and as a result, the server/load balancer will terminate the TLS handshake. On your Java app, this will manifest as one of the following error messages:
 

• javax.net.ssl.SSLHandshakeException: Received fatal alert: unknown_ca
• javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate

To resolve the problem, the operator needs to deploy the Diego Instance Root CA on the remote endpoint (server or load balancer), or disable the default client certificate, if mutual TLS is not required. 

It was mentioned above that most, but not all, outgoing HTTPS connections will automatically have the Diego Instance Id certificate included as the client's certificate. This happens when your connections are using the default SSLContext. If you were to modify the default SSLContext, default keystore/truststore, or if you were to create a custom SSLContext then you can bypass this behavior.

A notable case where this occurs is when using the MySQL JDBC driver. The JDBC driver creates its own SSLContext and so it does not inherit the automatic behavior enabled by the Java buildpack.