javax.net.ssl.SSLHandshakeException PKIX path building failed

Products

• CA Release Automation - Release Operations Center (Nolio)
• CA Release Automation - DataManagement Server (Nolio)
• CA Release Automation Connector

Issue/Introduction

While running actions that execute against HTTPS URL, the following error appears:

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target.

Environment

CA Release Automation Agent 6.x

Cause

The CA Release Automation Agent will generate the error message (also seen in the nolio_all.log) when it tries to access an HTTPS site that it does not have a certificate for (needed to establish a proper SSL handshake). 

Resolution

1. Download the certificate from the site in question.
2. If necessary, convert the certificate obtained in step #1 into an x509 format that can be imported into a java keystore (see Note below).
3. Import the certificate into the agents java keystore. You can do this by opening a command prompt on the artifact retrieval agent machine and:
   1. cd <NolioAgentInstallationFolder>
   2. jre/bin/keytool -importcert -file <fileFromStep2> -keystore jre/lib/security/cacerts -alias <aliasNameOfYourChoosing>
4. Restart the agent service.

NOTE: Regarding Step 2 (in the resolution section), please note the following keytool guidelines for importing certificates:

https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html#keytool_option_importcer

Additional Information

Additional Debugging/Troubleshooting:

This section will provide details on enabling more verbose diagnostic data for cases where additional troubleshooting is necessary. It will walk through enabling additional logging for an example scenario. Additional details of how Java's SSL works, tracing, etc.. can be found here: https://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/JSSERefGuide.html

Scenario:

In this scenario we are trying to look at the details behind the same PKIX error for this KB: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target.

In this case I believe I have already imported a certificate and want to look at the certificate being received by the secure endpoint. Using the certificate's "serial number" is a pretty decent way to compare what's being received vs what is in the truststore. There are a few key items that we'll find such as:

1. Some details for the certificate being offered by the secure endpoint. 
2. The truststore being used by the agent. 

 

Enable SSL Tracing On Agent:

1. Backup the conf/wrapper.conf file
2. Edit the wrapper.conf file to include the debug flag. Before doing this it is recommended to first identify which level of debugging you want to enable and then evaluate where is the best place to add the debug entry.
   Level of debugging: You can enable all debugging by using: -Djavax.net.debug=all
   Alternative level of debugging: Many variations of debugging exist. See the JSSERefGuide Debug section for details if you want to try troubleshooting it yourself without the all debug switch enabled. Logs sent to support should use "all". Example of subset/alternative options: -Djavax.net.debug=ssl,handshake,trustmanager
   Where to add the debug entry within the wrapper.conf: The wrapper.conf file has wrapper.java.additional.# entries. Use the next available number. For example, if you see "#wrapper.java.additional.5" then this line is considered commented out the next available number is 5 - as long as there are no other uncommented wrapper.java.additional.5 entries. 
   This is the next available number, on the test system used while stepping through the article, so the following entry is: wrapper.java.additional.5=-Djavax.net.debug=all
3. After saving the wrapper.conf and restarting the agent we run a test. 

 

The PKIX error appears in nolio_all.log AND nolio_action_exe.log. To begin understanding why:

• Open one of those files to find the pkix error. get the timestamp for the error.
• Start searching for that timestamp in the wrapper.log file.
  however, the timestamp captured by nolio_action_exe.log or nolio_all.log both separate seconds and milliseconds with a comma. the wrapper file separates it with a period. replace the comma with a period. Also, the year-month-day in nolio_action_exe|nolio_all are separated by a hyphen whereas the wrapper log separates with a forward slash. Replace the hyphens with forward slashes and begin searching for that timestamp. if you don't find an exact match you can start taking off numbers from the right (ms). 
• Next, search for:
  notepad++ regex search: Hello|TrustStore|cacerts|Certificate chain OR: 2019/08/07 21:51:58.*(Hello|TrustStore|cacerts|Certificate chain)
  linux/mac grep regex search: grep -E "Hello|TrustStore|cacerts|Certificate chain" wrapper.conf OR: grep -E "2019/08/07 21:51:58.*(Hello|TrustStore|cacerts|Certificate chain)" wrapper.conf

 

The wrapper.log does NOT log the truststore used each time the truststore is referenced. It shouldn't change between agent restarts. But it is an important piece of information. If the problem is intermittent, which might happen due to a load balanced secured endpoint returned a misconfigured server, then getting/keeping the truststore information is valuable. In the end, the successful/failed ssl handshakes is determined by which truststore the agent is using. All of this is being mentioned in case you know your agents log files only capture 2 days worth of data but it took 2 weeks to see the error. In this case it is recommended to make sure you capture the "trustStore is" message and then either:

• Save the file off someplace; or
• Increase logging for the wrapper.log within the conf/wrapper.conf file. There are two settings that can control this: 
  wrapper.logfile.maxsize=10m
  wrapper.logfile.maxfiles=5
  
  These are the defaults. Adjust accordingly.

The trustStore used by the Agent defaults to <AgentInstall>/jre/lib/security/cacerts. But it almost certainly can be changed (for example: possible via conf\wrapper.conf or via jre\lib\management\management.properties and possibly other ways too). Most of the areas where a alternative truststore can be specified would be done using a string similar to: -Djavax.net.ssl.trustStore=<path to truststore>

Once you've identified/confirmed the trustStore used by the agent then proceed identifying the problematic certificate received during/after the ServerHello exchange message. Do this by looking at the the details of the certificate in the "Certificate chain". The certificate with have a Subject, Signature, Validity Date, Issuer, and Serial Number as well as other information. But it is this information that you can use to verify whether that the certificate does exist in the truststore using the command:

<AgentInstallDir>\jre\bin\keytool -list -v -keystore <trustStore file location identified earlier>

From the keytool output you can search for the certificates serial number. It, as well as the other properties mentioned above, should be in the trustStore. If not then this error will occur. You may not be able to identify the server that is presenting this certificate. If it is a certificate that you are not expecting and the servername where the certificate from is not available then that portion of it will need to be investigated using network tool such as wireshark or router/switch/loadbalancer logs. 

  

Information for Technical Support:

- conf\wrapper.conf.

- jre\lib\management\management.properties.

- a copy of logs directory (with "all" tracing enabled).

- a copy of the truststore used.

 

There are other tools that can be used to gather more diagnostic data. For example: 

1. <AgentInstallDir>\jre\bin\keytool -printcert -sslserver <your ssl endpoint>:<port>
   This command will show the certificate being returned by a secure endpoint at that time. 
2. openssl s_client -showcerts -connect <your secured endpoint>:<port>
3. sslpoke: it seems there are several sources available. choose one at your discretion. This tool can test an ssl handshake with the secured endpoint while using a given truststore (by default: that version of java's jre/lib/security/cacerts - so be sure to either use the same java used by the agent or specify a specific truststore).