Client utilities are not communicating with application server on Primary server

Products

• CA Workload Automation AE - Business Agents (AutoSys)

Issue/Introduction

There was a database password change after which chk_auto_up failed with the following errors

chk_auto_up 
______________________________________________________________________________ 

CAUAJM_E_10029 Communication attempt with the CA WAAE Application Server has failed! [Primary:9000] 
CAUAJM_I_50054 Attempting (1) to Connect with Database: primarydb
CAUAJM_E_18407 Failed to connect to Oracle server: primarydb
CAUAJM_E_18400 An error has occurred while interfacing with ORACLE. 
CAUAJM_E_18401 Function <Olog> invoked from <openProc> failed <154> 
CAUAJM_E_18402 ORA-28001: the password has expired 

The password issue was resolved, but still remaining Communication issue with Applicationserver.

chk_auto_up
______________________________________________________________________________

CAUAJM_E_10029Communication attempt with the CA WAAE Application Server has failed! [Primary:9000]
CAUAJM_I_50054Attempting (1) to Connect with Database: primarydb
CAUAJM_I_50055*** Have Connected successfully with Database: primarydb. ***
______________________________________________________________________________

CAUAJM_I_50128Connected with Event Server: primarydb

______________________________________________________________________________

CAUAJM_I_50054Attempting (1) to Connect with Database: shadowdb
CAUAJM_I_50055*** Have Connected successfully with Database: shadowdb. ***
______________________________________________________________________________

CAUAJM_I_50128Connected with Event Server: shadowdb

______________________________________________________________________________

______________________________________________________________________________

CAUAJM_E_10029Communication attempt with the CA WAAE Application Server has failed! [Primary:9000]
CAUAJM_I_50038Checking CA WAAE Primary Scheduler on Machine: Primary
CAUAJM_I_50044Primary Scheduler is RUNNING on machine: Primary
______________________________________________________________________________

CAUAJM_I_50038Checking CA WAAE Shadow Scheduler on Machine: Shadow
CAUAJM_I_50045Shadow Scheduler is RUNNING on machine: Shadow
______________________________________________________________________________

CAUAJM_I_50038Checking CA WAAE Tie-Breaker Scheduler on Machine: Tiebreaker
CAUAJM_I_50047Tie-Breaker Scheduler is RUNNING on machine: Tiebreaker

 

Environment

Release : 11.3.6

Component : CA Workload Automation AE (AutoSys)

Cause

The application server has a problem in logging to the database primarydb

[04/23/2021 12:09:12]      CAUAJM_E_10656 The database primarydb has encountered a critical error. Please contact the database administrator.

The SSA does not listen on 9000 port on the primary but is working on the shadow  (cd /opt/CA/SharedComponents/Csam/SockAdapter/bin)

Primary AutoSys server
./csampmux status
Listeners

PID:11993212Port 49159 for IPv6.
PID:12124320Port 49161 for IPv6.
PID:11993212Port 49168 for IPv6.

Shadow AutoSys server  (cd /opt/CA/SharedComponents/Csam/SockAdapter/bin)
./csampmux status
Listeners

PID:24903714Port 9000 for IPv6.
PID:24903714Port 49211 for IPv6.
PID:14418144Port 49198 for IPv6.

Resolution

There was a hung application server process in the Primary server . After stopping it cleanly and killing the hung process the application sever was started.

Now it is working fine in Primary and Shadow server.