Autoping not working, logs show connection refused or Address already in use bind failed
search cancel

Autoping not working, logs show connection refused or Address already in use bind failed

book

Article ID: 200671

calendar_today

Updated On:

Products

CA Workload Automation AE - Business Agents (AutoSys) CA Workload Automation AE - System Agent (AutoSys) CA Workload Automation AE - Scheduler (AutoSys) Workload Automation Agent Autosys Workload Automation

Issue/Introduction

Autoping command fails and Agent's transmitter.log, show messages like below: 

01/01/2020 00:00:12.345-0400 5 TCP/IP Controller Plugin.Transmitter pool thread <Slow:1>.CybTargetHandlerChannel.call[:747] - Attempting to send:    20200925 12363847+0400 USER1234 . CONTROL ACTIVE OSText(Linux for amd64) ByUser(rootuser) AgentRelease(12.0)  AgentBuild(1609, Service Pack 00, Maintenance Level 0)  User(USER1234) Host(server.example.com)
01/01/2020 00:00:12.345-0400 5 TCP/IP Controller Plugin.Transmitter pool thread <Slow:1>.CybTargetHandlerChannelLogHelper.logConnectionAttempt[:75] - Attempting to open conversation to [email protected]:7612 using plain socket
01/01/2020 00:00:12.345-0400 1 TCP/IP Controller Plugin.Transmitter pool thread <Slow:1>.CybTargetHandlerChannel.a[:1202] - Error connecting to ACE_SCH:                                                                                            cybermation.library.communications.CybConversationConnectBindException: Address already in use (Bind failed)

 

Environment

Workload Automation System Agent

Cause

  1. Ports to Application server are not open from Agent.  If Application server is on port 7500, then that port must be open from Agent host.
  2. Usually this is noted to occur when all the ephemeral ports are in use or in a pending state (CLOSE_WAIT, etc). So, check if the OS has any ephemeral ports left or the agent to use, on the Agent host.

Resolution

  1. Ensure autosys application server and scheduler ports are open bi-directional, from the Agent host.
  2. Likely Agent host ran out out of ephemeral ports at that time.  When the agent needs to connect to a server port, it needs a temporary port from the system as well. When the ephemeral port is not available, you will get that error message in the transmitter.log.  To resolve this, a restart of the agent process or a reboot of the Agent host itself maybe needed.
  3. Agent may not respond to stop requests too. See below use case:

./cybAgent -s

## there after output of  netstat -anp showed that there were 28k processes with CLOSE_WAIT for port 35004 help by PID 9667 for java. This is perhaps not allowing the agent to get any more ephemeral ports.

tcp6       0      0 x.x.x.x:59727    x.x.x.x:35004    CLOSE_WAIT  9667/java           
tcp6       0      0 x.x.x.x:53710    x.x.x.x:35004    CLOSE_WAIT  9667/java           
tcp6       0      0 x.x.x.x:46768    x.x.x.x:35004    CLOSE_WAIT  9667/java           
tcp6       0      0 x.x.x.x:47232    x.x.x.x:35004    CLOSE_WAIT  9667/java           
tcp6       0      0 x.x.x.x:60299    x.x.x.x:35004    CLOSE_WAIT  9667/java          

 

## kill the PID directly
kill -9  9667

## Then restart the agent now

Powered by Wolken Software