Error: Connection closed by the other side causing Tomcat deadlock - AutoSys WCC

Products

• CA Workload Automation AE - Scheduler (AutoSys)

Issue/Introduction

AutoSys Web UI (WCC) services remain in an active state according to the operating system, but the user interface on port 8443 is inaccessible. This behavior typically follows a network disruption where the Hazelcast cluster connection on port 7004 is forcibly terminated by external security software, resulting in a "split-brain" event in the underlying Hazelcast cluster, which deadlocks the Tomcat HTTP thread pool.

Environment

Product: AutoSys Workload Control Center (WCC)
Product Version: ALL
Platform: Linux (SLES 15, RHEL)
Security Software: Trend Micro Deep Security, CrowdStrike, Firewalls

Cause

Security monitors or "Intrusion Prevention" tools like Trend Micro intercept long-lived TCP sessions on port 7004. Forcibly closing these connections triggers a "split-brain" event in the underlying Hazelcast cluster, which deadlocks the Tomcat HTTP thread pool.

Symptom: The CA-wcc.log contains the following log entries before the interface becomes unresponsive:

INFO | jvm 1 | ####/##/## ##:##:## | ... closed. Reason: Connection closed by the other side
INFO | jvm 1 | ####/##/## ##:##:## | ... Removed connection to endpoint: [hostname]:7004

• Port 8443 is not listening or returns Connection Refused
• Port 7004 remains in a LISTEN state.
• systemctl status ca-wcc shows the service as active (running).

 

Resolution

To resolve this issue and prevent recurrence, implement the following configuration changes:

1. Security Software Whitelisting:
   • Whitelist TCP ports 7004 and 8443 for bidirectional traffic between all cluster nodes.
   • Configure security software to exclude the WCC installation directory (default: /opt/CA/WorkloadAutomationAE/wcc/) from active scanning. 
     
     
2. Harden Tomcat Configuration:
   • Navigate to $CA_WCC_INSTALL_LOCATION/tomcat/conf/.
   • Open server.xml.
   • Locate the line <Server port="8005" shutdown="SHUTDOWN">.
   • Change the port to -1: <Server port="-1" shutdown="SHUTDOWN">.
     
     
3. Service Restart:
   • Restart the WCC services to clear the deadlocked threads: sudo systemctl restart ca-wcc

To speak with a customer representative or a Support Engineer, see Contact Support. Scroll to the bottom of the page and click on your respective region.

 

Additional Information

If the issue persists, capture a thread dump using jstack -l [PID] > /tmp/wcc_deadlock.txt and provide it to support.

For more information on WCC cluster requirements, see WCC Cluster Formation Issues