Error: Intermittent AwE-5103 network socket error (Connection reset) on established Agent connections

Products

• CA Automic Applications Manager (AM)

Issue/Introduction

This article addresses cases where an AppWorx Agent, which has been running successfully, suddenly disconnects with an AwE-5103 error. Unlike startup failures, these are intermittent drops of an active connection.

Symptoms

• Agent intermittently moves to SRVC_DOWN or STOPPED status.
• Agent logs show that logging abruptly stops
• RmiServer logs show: AwE-5103 network socket error: java.net.SocketException: Connection reset.
• The issue may be resolved temporarily by restarting the Agent service or the host server.

Environment

CA Automic Applications Manager 9.x.x

Cause

A "Connection reset" on an established socket indicates that the local OS received a TCP RST packet. This is typically not an AppWorx software defect, but rather an external interruption:

1. Stateful Firewall/Load Balancer Timeouts: A network device may have silently dropped the connection state, sending a RST when the next packet is sent.
2. Deep Packet Inspection (DPI): Security appliances may terminate long-lived SSL/TLS sessions they deem suspicious.
3. TCP Chimney/Offload Issues: (Windows Specific) Network card features may incorrectly handle the offloading of long-lived sockets.
4. OS Resource Exhaustion: The OS may close sockets if it runs out of ephemeral ports or system handles.

Resolution

To identify the source of the reset, perform the following:

1. Identify the RST Origin: Use a network capture tool (e.g., Wireshark or tcpdump) on both the Master and Agent. Check which side originates the RST packet.
2. Check Firewall Persistent Settings: Ensure that any firewalls between the Master and Agent have 'TCP Keep-Alive' enabled and that idle timeouts exceed the AppWorx polling interval.
3. Disable NIC Offloading: On Windows Agents, try disabling 'TCP Chimney Offload' and 'Receive Side Scaling' (RSS) via the Network Adapter properties to see if stability improves.
4. Monitor OS Sockets: Run netstat -ano | findstr <Port> during the failure to see if the socket is in CLOSE_WAIT, which indicates the other side closed the connection first.

Additional Information

If the issue is resolved by a server reboot, it likely indicates that the OS network stack or a security driver was intercepting the traffic.

See Contact Support to engage a Network Engineer for log analysis.