Java Agents cannot reconnect to AE - java.io.IOException: Too many open files

Products

• CA Automic Workload Automation - Automation Engine

Issue/Introduction

After a deconnection of the JCP, the Java Based Agents (Agent JMX, Agent PeopleSoft, Agent RA Core, Agent SAP, Agent SQL, Analytics.Backend) have a file descriptor leak, as they don't handle the disconnection from the JCP well.

This causes the process thread count to increase and can lead to reaching the user thread count limit. The thread count can be checked with a ps command of lsof.

If the limit is reached the following error appears in the Agent log:

20220831/044035.198 - 1      com.uc4.ex.cp.InitialConnectionException: Initial connection with endpoint not possible. Please check your configuration.
20220831/044035.199 - 1      U02000379 Initiating connection to server 'AE:443' using WebSocket URI: 'wss://AE:443/agent'.
20220831/044035.212 - 1      java.io.IOException: Too many open files

Environment

Release : 21.0.x

Component(s): Agent JMX, Agent PeopleSoft, Agent RA Core, Agent SAP, Agent SQL, Analytics.Backend

Cause

Defect: failed connection attempts to the JCP resulted in a connection-leak.

Resolution

Workaround:

To avoid the agent being unable to connect to the JCP, increase the maximum open file limits for the user who starts the agent (by default on Linux is 1024).

This can be increased either in the .profile (not recommended) of the user or in the /etc/security/limits.conf with this kind of lines (replacing <USER> by the user starting the Agent):

<USER> soft nofile 4096
<USER> hard nofile 8192

Solution:

Update to a fix version listed below or a newer version if available.

Fix version:
Component(s): Agent JMX, Agent PeopleSoft, Agent RA Core, Agent SAP, Agent SQL, Analytics.Backend

Automation.Engine 21.0.4 HF1 - Available
Automation.Engine 21.0.5 - Available

Additional Information

Important Updates regarding the fixing of this bug:

During the fix of this initial bug, the rest of the Java-based components like TLS GW, Proxy, and other components which also use this functionality, were not selected as modules requiring this correction.
These omitted but still affected Java-based components will be re-built and delivered shortly in the next AE Service Pack.

Note that if you are running the affected Java agent on Open Java v1.8, which is not supported, the problem remains. You will have to upgrade to OpenJDK v11 or Oracle Java 1.8 to prevent the issue from happening.