Error: CAUAJM_E_10436 after the EEM HA failover. WAAE CLIs does not work.

Products

CA Workload Automation AE - Scheduler (AutoSys) Autosys Workload Automation

Issue/Introduction

Workload Automation AE Commands fail with the following message soon after the application server restart:

[autosys@waaeserver1 ~]$ autosyslog -e
CAUAJM_E_10436 Security server unreachable or invalid authentication certificate file.
CAUAJM_E_10434 Error initiating security session.
[EE_BADOBJECT Bad Object]
[ISP_ERROR_NOGATEWAY igateway not running]
[Authenticate Error: Authentication Failed]
[Identity Attempted: ]
[CertificateReader::loadPEM - cannot read certificate]

CAUAJM_W_10406 Control Execute Access Denied!
CAUAJM_E_10434 Error initiating security session.
CAUAJM_W_10440 Class: as-control Resource: ACE.EPLOG User: autosys Access: execute
CAUAJM_W_10442 Time: 1455405684 Delegator: None

[autosys@waaeserver1 ~]$ autorep -J ALL
/*CAUAJM_E_10436 Security server unreachable or invalid authentication certificate file.
CAUAJM_E_10434 Error initiating security session.
[EE_BADOBJECT Bad Object]
[ISP_ERROR_NOGATEWAY igateway not running]
[Authenticate Error: Authentication Failed]
[Identity Attempted: ]
[CertificateReader::loadPEM - cannot read certificate]

*/

The Application Server log ($AUTOUSER/out/as_server.$AUTOUSER)repeatedly logs the below message soon after its restart:

[02/14/2016 04:49:06]      CAUAJM_I_40275 Log Rollover level set to <MIDNIGHT,SIZE(100),PURGE(7)>.
[02/14/2016 04:49:06]      CAUAJM_I_40244 EnableIPCaching value set to <0>.
[02/14/2016 04:49:06]      CAUAJM_I_40244 LogMaxEndLines value set to <0>.
[02/14/2016 04:49:06]      CAUAJM_I_40211 Using TZ = Asia/Kolkata.
[02/14/2016 04:49:06]      CAUAJM_I_10655 System is running in single server mode. Event server: autosysdb.
[02/14/2016 04:49:11]      CAUAJM_I_20197 CA WAAE Application Server operational on port 9000.
[02/14/2016 04:49:11]      CAUAJM_I_40244 CA EEM unauthenticated user mode value set to <**UNKNOWN**>.
[02/14/2016 04:49:11]      CAUAJM_I_20366 CA WAAE Application Server operational on agent listener port 49156.
[02/14/2016 04:49:11]      CAUAJM_I_20367 CA WAAE Application Server operational on auxiliary agent listener port 7500.
ERROR: [0x036ccb70] ispUtil::TcpConnect: Error in delayed connection() 111 - Connection refused
ERROR: [0x036ccb70] ispUtil::TcpConnect Failed to connect to host [eemserver1]
ERROR: [0x036ccb70] ispUtil.HttpPostRequest: ObtainPC Failed
log4cxx: No appender could be found for logger (PozFactory).
log4cxx: Please initialize the log4cxx system properly.
ERROR: [0x036ccb70] ispUtil::TcpConnect: Error in delayed connection() 111 - Connection refused
ERROR: [0x036ccb70] ispUtil::TcpConnect Failed to connect to host [eemserver1]
ERROR: [0x036ccb70] ispUtil.HttpPostRequest: ObtainPC Failed
....
...
..
ERROR: [0x036ccb70] ispUtil::TcpConnect: Error in delayed connection() 111 - Connection refused
ERROR: [0x036ccb70] ispUtil::TcpConnect Failed to connect to host [eemserver1]
ERROR: [0x036ccb70] ispUtil.HttpPostRequest: ObtainPC Failed
ERROR: [0x036ccb70] CertificateReader::loadPEM : etpki_pem_file_to_cert failed [ errorcode : -1 ]
[02/14/2016 04:49:12]      CAUAJM_E_10436 Security server unreachable or invalid authentication certificate file.
[02/14/2016 04:49:12]      CAUAJM_E_10434 Error initiating security session.
[02/14/2016 04:49:12]      CAUAJM_E_10437 Detailed Error Information:
[02/14/2016 04:49:12]      [EE_BADOBJECT Bad Object]
[02/14/2016 04:49:12]      [ISP_ERROR_NOGATEWAY igateway not running]
[02/14/2016 04:49:12]      [Authenticate Error: Authentication Failed]
[02/14/2016 04:49:12]      [Identity Attempted: ]
[02/14/2016 04:49:12]      [CertificateReader::loadPEM - cannot read certificate]
ERROR: [0x03158b70] ispUtil::TcpConnect: Error in delayed connection() 111 - Connection refused ERROR: [0x03158b70] ispUtil::TcpConnect Failed to connect to host [eemserver1] ERROR: [0x03158b70] ispUtil.HttpPostRequest: ObtainPC Failed .... ... ..

Environment

Release: All supported and EEM Multi-Write configured environments
Component: ATSEEM

Cause

EEM is configured in HA (a.k.a Multi-write)
WAAE external security is not configured with EEM server-list as per EEM HA setup
WAAE Application Server is restarted when the Primary EEM is down

Resolution

Configure the WAAE security to work with all nodes of the EEM cluster.

Ensure all EEM clustered servers are up and running.
Using the ‘autosys_secure’ utility set the EEM server location and regenerate the certificate by adding all nodes of EEM cluster with comma separated values.
WAAE 11.3.6 SP2 is used in the below scenario.

[autosys@waaeserver1 ~]$ autosys_secure

CA WAAE Security Utility

Please select from the following options:
[1] Revert to NATIVE instance security.
[2] Manage CA EEM security settings.
[3] Change database password.
[4] Change remote authentication method.
[5] Manage user@host or user@domain users.
[6] Get encrypted password.
[0] Exit CA WAAE Security Utility.
> 2

Manage CA EEM security settings

Please select from the following options:
[1] Manage CA EEM server settings.
[2] Manage cached credentials.
[9] Exit from "Manage CA EEM security settings" menu.
[0] Exit CA WAAE Security Utility.
> 1

Manage CA EEM server settings

Please select from the following options:
[1] Show current CA EEM server settings.
[2] Set CA EEM server location and regenerate certificate.
[3] Set unauthenticated user mode.
[9] Exit from "Manage CA EEM server settings" menu.
[0] Exit CA WAAE Security Utility.
> 1

CAUAJM_I_60228 CA EEM server: eemserver1
CAUAJM_I_60342 Unauthenticated user mode: OFF

Please select from the following options:
[1] Show current CA EEM server settings.
[2] Set CA EEM server location and regenerate certificate.
[3] Set unauthenticated user mode.
[9] Exit from "Manage CA EEM server settings" menu.
[0] Exit CA WAAE Security Utility.
> 2
Input the CA EEM server name(s) (or hit enter to cancel): eemserver1,eemserver2
Input the CA EEM administrator name (or hit enter to cancel): EiamAdmin
Input the CA EEM administrator password:

Confirm the CA EEM administrator password:

CAUAJM_I_60200 CA EEM certificate generated successfully.
CAUAJM_I_60191 The CA EEM server location was changed successfully.

Please select from the following options:
[1] Show current CA EEM server settings.
[2] Set CA EEM server location and regenerate certificate.
[3] Set unauthenticated user mode.
[9] Exit from "Manage CA EEM server settings" menu.
[0] Exit CA WAAE Security Utility.
> 0

A quick test after security changes:

WAAE CLIs no longer fail.
While the EEM primary is down, restart the application server and monitor the log ($AUTOUSER/out/as_server.$AUTOSERV)for the following message:

[02/14/2016 06:59:13]      CAUAJM_I_20367 CA WAAE Application Server operational on auxiliary agent listener port 7500.
ERROR: [0x07371b70] ispUtil::TcpConnect: Error in delayed connection() 111 - Connection refused
ERROR: [0x07371b70] ispUtil::TcpConnect Failed to connect to host [eemserver1]
ERROR: [0x07371b70] ispUtil::TcpConnect: Error in delayed connection() 111 - Connection refused
ERROR: [0x07371b70] ispUtil::TcpConnect Failed to connect to host [eemserver1]
ERROR: [0x07371b70] ispUtil.HttpPostRequest: ObtainPC Failed
log4cxx: No appender could be found for logger (PozFactory).
log4cxx: Please initialize the log4cxx system properly.
CAUAJM_I_10485 CA EEM security session initialized with server <eemserver1,eemserver2>.
[02/14/2016 06:59:14]      CAUAJM_W_10472 Resource manager encountered the following during initialization: <Library "libDCAMClient" has not been loaded>.
[02/14/2016 06:59:14]      CAUAJM_W_10473 Resource manager will ignore non-virtual resource constraints for all jobs.
[02/14/2016 06:59:14]      CAUAJM_I_10474 Resource manager initialization complete.
[02/14/2016 06:59:15]      CAUAJM_I_30001 CA WAAE Application Server startup complete.

Additional Information

EEM Failover Configuration

WAAE autosys_secure utility