Agent Connection Problems: General SSLEngine problem

Products

CA Release Automation - Release Operations Center (Nolio) CA Release Automation - DataManagement Server (Nolio)

Issue/Introduction

AIX Agents lose connectivity with their Execution Server. In the agent's nimi.log file the following exception can be found.

javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at com.ibm.jsse2.ib.A(ib.java:350)
at com.ibm.jsse2.SSLEngineImpl.b(SSLEngineImpl.java:28)
at com.ibm.jsse2.SSLEngineImpl.a(SSLEngineImpl.java:479)
at com.ibm.jsse2.SSLEngineImpl.unwrap(SSLEngineImpl.java:529)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:24)
at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:868)
at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:605)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:282)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:216)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:281)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:201)
at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:949)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:973)
at java.lang.Thread.run(Thread.java:767)
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at com.ibm.jsse2.p.a(p.java:10)
at com.ibm.jsse2.SSLEngineImpl.a(SSLEngineImpl.java:122)
at com.ibm.jsse2.ib.a(ib.java:319)
at com.ibm.jsse2.ib.a(ib.java:437)
at com.ibm.jsse2.jb.a(jb.java:565)
at com.ibm.jsse2.jb.a(jb.java:25)
at com.ibm.jsse2.ib.s(ib.java:582)
at com.ibm.jsse2.ib$1.run(ib$1.java:2)
at java.security.AccessController.doPrivileged(AccessController.java:448)
at com.ibm.jsse2.ib$c_.run(ib$c_.java:5)
at org.jboss.netty.handler.ssl.SslHandler$2.run(SslHandler.java:989)
at org.jboss.netty.handler.ssl.ImmediateExecutor.execute(ImmediateExecutor.java:37)
at org.jboss.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:986)
at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:886)
... 12 more
Caused by: com.ibm.jsse2.util.h: No trusted certificate found
at com.ibm.jsse2.util.g.a(g.java:35)
at com.ibm.jsse2.util.g.b(g.java:74)
at com.ibm.jsse2.util.e.a(e.java:22)

Environment

Release : 6.6

Component : CA RELEASE AUTOMATION EXECUTION SERVER

Cause

A certificate, alias: nolio, for CA Release Automation (CARA) with validity of 10 years had expired on November 26, 2020. This key/alias is found in the conf/nolio.jks file.

The nolio.jks, it contains the following list of aliases:

nolio - It expired on 2020-11-26. It is used for:
- Used by nimi when it is configured to communicate securely.
- Used when Single Sign-On has been enabled. For more details related to this issue please see KB Article 204136
nolio_old - is not used. It expired on 22.09.2010.
nolioserver - is used for Secure UI Communication and Secure Management Server to Execution Server Communication. It will expire on 30.11.2026. More details on these two configurations can be found here: Secure Communications

NOTE:

Based on our investigation we have found that it is possible for the certificate in the keyStore.jks (on execution server or agents) to also be expired - if the execution server or agent was installed more than 10 years ago. So it is worth taking this into consideration when evaluating potential Execution Server <-> Agent connectivity problems related to SSL.

All of the occurrences we have seen have not involved an expired certificate in the keyStore.jks file. The occurrences we have seen involved the expired nolio alias as it is used while validating the Chain of Trust on AIX agents.

This problem appears to be more pronounced on AIX agents than it is on Windows or Linux. Many/most Linux Nolio Agents are installed with a packaged/embedded version of Oracle Java while Nolio Agents on AIX use a preinstalled IBM packaged version of Java which appears to use a different TrustManager. Oracle Java versions appear to be not as strict when it comes to validating a certificates certificate chain (or Chain of Trust). If you are having certificate problems on Linux or Windows than it is likely not due to the expired nolio certificate, but is instead likely related to the certificate in the keyStore.jks file having expired.

Resolution

There are three options for addressing this problem. One is a solution. The other is a workaround.

Solution #1

Apply 6.6.8 or 6.7.3 - which includes an updated/unexpired certificate.

New certificates for 6.7 are being prepared to be delivered via the next 6.7 cumulative patch (aka 6.7.3). This patch is available. Its Release Notes can be found here: Release Notes for 6.7.3
New certificates for 6.6 are being prepared to be delivered via the next 6.6 cumulative patch (aka 6.6.8). This patch is available. Its Release Notes can be found here: Release Notes for 6.6.8

Important

If your NES (Execution Server) and Agents are communicating via SSL, it is extremely important to review the following KB Article before applying 6.6.8 or 6.7.3. Applying the cumulative will break the connection between Agents and NES if you do not take the appropriate steps (on both agents and NES) to disable SSL Connectivity temporarily. KB Article: Nolio Agents Unreachable After 6.7.0.b398

Solution #2

Implement your own signed certificates for Execution Servers and Agents to communicate securely. The details for this are outlined here: Secure Communications

However, please note that the NES and Agent will not be able to communicate if they are configured to communicate via SSL and have certificates that they do not trust. Until the NES and Agent have certificates trusted by each side, both the NES and Agent should be configured to communicate without SSL. See the following KB Article for an example and additional details: Nolio Agents Unreachable After 6.7.0.b398

Workaround

Temporarily disable secure communications between the Execution Server and Agents experiencing this problem, until new certificates are applied.

On the Execution Server involved in these connection errors, backup your conf/nimi_config.xml.
Update the conf/nimi_config.xml on the targeted Execution Server. Specifically, its: /config/nimi/network/security/enabled from true to false.
Restart the Execution Server service.