Enabling debugging of SSL connections from the proxy-engine to the backend server in Access Gateway (formerly CA Secure Proxy Server)

Products

• CA Single Sign On Secure Proxy Server (SiteMinder)
• CA Single Sign On SOA Security Manager (SiteMinder)
• CA Single Sign-On

Issue/Introduction

The Access Gateway (formerly CA Secure Proxy Server) is used as a reverse proxy to connect users via the SPS to backend servers.  The connections from SPS to the backend servers will often be over SSL (i.e. accessed using HTTPS:).  This article shows how to enable debug logging so that problems can be identified and resolved.

 

 

 

Environment

Environment: All (Windows, Linux)

Component: SMSPS

Cause

There can be a number of issues establishing a SSL connection from proxy-engine to the backend, there are some restrictions on the level of cryptographic that java is able to use; there can be different levels of SSL/TLS version supported by the backend server; and there can be issues with the trust path for the backend server certificates.  

Resolution

Instructions: 

Java provides a handy setting that will log detail of the SSL handshake and transferred data  into stdout by adding the parameter -Djavax.net.debug=all to the java runtime startup. 

 

Enable the debug SSL in SPS  

For SPS the files to apply that debug setting are : 

• Windows : proxy-engine/conf/SmSpsProxyEngine.properties
• Unix  : proxy-engine/proxyserver.sh

And we need to add the parameter -Djavax.net.debug=all to the java startup command as pictured below for each environment.

 

(Note: Please make sure you edit the correct one for your environment; for Windows you will need to stop the proxy-engine service before it will allow you to save the .properties file)

The proxy-engine service needs to be restarted for the setting to take effect. 

 

Review the SSL trace details in the nohup log

Once the service is restarted trace information for any SSL connection from the proxy-engine to the backend is written to the timestamped nohup*.out file.

In the following we can see details of the SSL ClientHello message send to the backend server. 

Particular details to look for in the logs are:

• The TLS versions and acceptable cipher suites send from the proxy-engine to the backend in the SSL ClientHello message
• The cipher suite chosen by the backend server in the returned SSL ServerHello message.
• The details of the backend server's X.509 certificate also in the SSL ServerHello message
• Details for tracing the trust certificate chain for the backend server's certificate, as provided in the SSL ServerHello and the locally loaded CA Certificates.
• The encryption and digest for the first message send from the proxy-engine to the backend server after the handshake (A failure at this point can indicate that the java JVM has export limited cryptography settings - see java unlimited jurisdiction policy files note at the end )

 

Additional Information

Java SE Debugging SSL/TLS Connections 
https://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/ReadDebug.html

Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files 8 Download
https://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html

Details about SSL/TLS
https://en.wikipedia.org/wiki/Transport_Layer_Security