Error: CAUAJM_E_10436 after the EEM HA failover. WAAE CLIs does not work.

book

Article ID: 38040

calendar_today

Updated On:

Products

CA Workload Automation AE - Business Agents (AutoSys) CA Workload Automation AE - Scheduler (AutoSys) CA Workload Automation Agent

Issue/Introduction

Symptoms:

  Workload Automation AE Commands fail with the following message soon after the application server restart:

[[email protected] ~]$ autosyslog -e
CAUAJM_E_10436 Security server unreachable or invalid authentication certificate file.
CAUAJM_E_10434 Error initiating security session.
[EE_BADOBJECT Bad Object]
[ISP_ERROR_NOGATEWAY igateway not running]
[Authenticate Error: Authentication Failed]
[Identity Attempted: ]
[CertificateReader::loadPEM  - cannot read certificate]

CAUAJM_W_10406 Control Execute Access Denied!
CAUAJM_E_10434 Error initiating security session.
CAUAJM_W_10440 Class: as-control Resource: ACE.EPLOG User: autosys Access: execute
CAUAJM_W_10442 Time: 1455405684  Delegator: None

[[email protected] ~]$ autorep -J ALL
/*CAUAJM_E_10436 Security server unreachable or invalid authentication certificate file.
CAUAJM_E_10434 Error initiating security session.
[EE_BADOBJECT Bad Object]
[ISP_ERROR_NOGATEWAY igateway not running]
[Authenticate Error: Authentication Failed]
[Identity Attempted: ]
[CertificateReader::loadPEM  - cannot read certificate]

*/

  The Application Server log ($AUTOUSER/out/as_server.$AUTOUSER)repeatedly logs the below message soon after its restart:

[02/14/2016 04:49:06]      CAUAJM_I_40275 Log Rollover level set to <MIDNIGHT,SIZE(100),PURGE(7)>.
[02/14/2016 04:49:06]      CAUAJM_I_40244 EnableIPCaching value set to <0>.
[02/14/2016 04:49:06]      CAUAJM_I_40244 LogMaxEndLines value set to <0>.
[02/14/2016 04:49:06]      CAUAJM_I_40211 Using TZ = Asia/Kolkata.
[02/14/2016 04:49:06]      CAUAJM_I_10655 System is running in single server mode.  Event server:  autosysdb.
[02/14/2016 04:49:11]      CAUAJM_I_20197 CA WAAE Application Server operational on port 9000.
[02/14/2016 04:49:11]      CAUAJM_I_40244 CA EEM unauthenticated user mode value set to <**UNKNOWN**>.
[02/14/2016 04:49:11]      CAUAJM_I_20366 CA WAAE Application Server operational on agent listener port 49156.
[02/14/2016 04:49:11]      CAUAJM_I_20367 CA WAAE Application Server operational on auxiliary agent listener port 7500.
ERROR: [0x036ccb70] ispUtil::TcpConnect: Error in delayed connection() 111 - Connection refused
ERROR: [0x036ccb70] ispUtil::TcpConnect Failed to connect to host [eemserver1]
ERROR: [0x036ccb70] ispUtil.HttpPostRequest: ObtainPC Failed
log4cxx: No appender could be found for logger (PozFactory).
log4cxx: Please initialize the log4cxx system properly.
ERROR: [0x036ccb70] ispUtil::TcpConnect: Error in delayed connection() 111 - Connection refused
ERROR: [0x036ccb70] ispUtil::TcpConnect Failed to connect to host [eemserver1]
ERROR: [0x036ccb70] ispUtil.HttpPostRequest: ObtainPC Failed
....
...
..
ERROR: [0x036ccb70] ispUtil::TcpConnect: Error in delayed connection() 111 - Connection refused
ERROR: [0x036ccb70] ispUtil::TcpConnect Failed to connect to host [eemserver1]
ERROR: [0x036ccb70] ispUtil.HttpPostRequest: ObtainPC Failed
ERROR: [0x036ccb70] CertificateReader::loadPEM : etpki_pem_file_to_cert failed [ errorcode : -1 ]
[02/14/2016 04:49:12]      CAUAJM_E_10436 Security server unreachable or invalid authentication certificate file.
[02/14/2016 04:49:12]      CAUAJM_E_10434 Error initiating security session.
[02/14/2016 04:49:12]      CAUAJM_E_10437 Detailed Error Information:
[02/14/2016 04:49:12]      [EE_BADOBJECT Bad Object]
[02/14/2016 04:49:12]      [ISP_ERROR_NOGATEWAY igateway not running]
[02/14/2016 04:49:12]      [Authenticate Error: Authentication Failed]
[02/14/2016 04:49:12]      [Identity Attempted: ]
[02/14/2016 04:49:12]      [CertificateReader::loadPEM  - cannot read certificate]
ERROR: [0x03158b70] ispUtil::TcpConnect: Error in delayed connection() 111 - Connection refused
ERROR: [0x03158b70] ispUtil::TcpConnect Failed to connect to host [eemserver1]
ERROR: [0x03158b70] ispUtil.HttpPostRequest: ObtainPC Failed
....
...
..

Environment:

  All supported and EEM Multi-Write configured environments

Cause:

  • EEM is configured in HA (a.k.a Multi-write)
  • WAAE external security is not configured with EEM server-list as per EEM HA setup
  • WAAE Application Server is restarted when the Primary EEM is down

Resolution: 

  Configure the WAAE security to work with all nodes of the EEM cluster.

  • Ensure all EEM clustered servers are up and running.
  • Using the  ‘autosys_secure’ utility set the EEM server location and regenerate the certificate by adding all nodes of EEM cluster with comma separated values.
    WAAE 11.3.6 SP2 is used in the below scenario.

[[email protected] ~]$ autosys_secure

CA WAAE Security Utility


Please select from the following options:
[1] Revert to NATIVE instance security.
[2] Manage CA EEM security settings.
[3] Change database password.
[4] Change remote authentication method.
[5] Manage [email protected] or [email protected] users.
[6] Get encrypted password.
[0] Exit CA WAAE Security Utility.
>  2

Manage CA EEM security settings


Please select from the following options:
[1] Manage CA EEM server settings.
[2] Manage cached credentials.
[9] Exit from "Manage CA EEM security settings" menu.
[0] Exit CA WAAE Security Utility.
>  1

Manage CA EEM server settings


Please select from the following options:
[1] Show current CA EEM server settings.
[2] Set CA EEM server location and regenerate certificate.
[3] Set unauthenticated user mode.
[9] Exit from "Manage CA EEM server settings" menu.
[0] Exit CA WAAE Security Utility.
>  1

CAUAJM_I_60228 CA EEM server:  eemserver1
CAUAJM_I_60342 Unauthenticated user mode: OFF


Please select from the following options:
[1] Show current CA EEM server settings.
[2] Set CA EEM server location and regenerate certificate.
[3] Set unauthenticated user mode.
[9] Exit from "Manage CA EEM server settings" menu.
[0] Exit CA WAAE Security Utility.
>  2
Input the CA EEM server name(s) (or hit enter to cancel):  eemserver1,eemserver2
Input the CA EEM administrator name (or hit enter to cancel):  EiamAdmin
Input the CA EEM administrator password:

Confirm the CA EEM administrator password:


CAUAJM_I_60200 CA EEM certificate generated successfully.
CAUAJM_I_60191 The CA EEM server location was changed successfully.


Please select from the following options:
[1] Show current CA EEM server settings.
[2] Set CA EEM server location and regenerate certificate.
[3] Set unauthenticated user mode.
[9] Exit from "Manage CA EEM server settings" menu.
[0] Exit CA WAAE Security Utility.
>  0

A quick test after security changes:

  • WAAE CLIs no longer fail.
  • While the EEM primary is down, restart the application server and monitor the log ($AUTOUSER/out/as_server.$AUTOSERV)for the following message:

[02/14/2016 06:59:13]      CAUAJM_I_20367 CA WAAE Application Server operational on auxiliary agent listener port 7500.
ERROR: [0x07371b70] ispUtil::TcpConnect: Error in delayed connection() 111 - Connection refused
ERROR: [0x07371b70] ispUtil::TcpConnect Failed to connect to host [eemserver1]
ERROR: [0x07371b70] ispUtil::TcpConnect: Error in delayed connection() 111 - Connection refused
ERROR: [0x07371b70] ispUtil::TcpConnect Failed to connect to host [eemserver1]
ERROR: [0x07371b70] ispUtil.HttpPostRequest: ObtainPC Failed
log4cxx: No appender could be found for logger (PozFactory).
log4cxx: Please initialize the log4cxx system properly.
CAUAJM_I_10485 CA EEM security session initialized with server <eemserver1,eemserver2>.
[02/14/2016 06:59:14]      CAUAJM_W_10472 Resource manager encountered the following during initialization: <Library "libDCAMClient" has not been loaded>.
[02/14/2016 06:59:14]      CAUAJM_W_10473 Resource manager will ignore non-virtual resource constraints for all jobs.
[02/14/2016 06:59:14]      CAUAJM_I_10474 Resource manager initialization complete.
[02/14/2016 06:59:15]      CAUAJM_I_30001 CA WAAE Application Server startup complete.

 

Additional Information:

EEM Failover Configuration

WAAE autosys_secure utility

Environment

Release:
Component: ATSEEM