EEM Cluster Primary, Secondary Synchronization Problem SafeException: EE_SPONSORERROR

Products

• CA Workload Automation AE - Scheduler (AutoSys)

Issue/Introduction

 After adding a secondary to the primary EEM server using eiam-clustersetup utility, the sync command from the secondary fails with an error:

C:\Program Files (x86)\CA\SC\EmbeddedEntitlementsManager\bin>java -jar eiam-clustersetup.jar -p eemserver1.cluster.com
Feb 2, 2016 1:45:16 PM IclUtil itechLibInit
INFO: iTechSDK initialized successfully
INFO  - EIAM_HOME [C:\Program Files (x86)\CA\SC\EmbeddedEntitlementsManager\]
INFO  - IGW_LOC [C:\Program Files (x86)\CA\SC\iTechnology\]
INFO  - DXHOME [C:\Program Files (x86)\CA\Directory\dxserver/]
INFO  - Hostname identified as [eemserver2.example.com]
EiamAdmin password :
INFO  - Checking server status
INFO  - igateway status      [started]
INFO  - dxserver status      [started]

Are you sure you want to continue? [Y/N]:Y
[eemserver2.example.com]>sync
=======================================================
INFO  - Select current machine hostname
=======================================================
[Authenticate Error: Authentication Failed, Identity Attempted: EiamAdmin]
com.ca.eiam.SafeException: EE_SPONSORERROR iSponsor Error
        at com.ca.eiam.SafeContext.authenticateWithPassword(SafeContext.java:1683)
        at com.ca.eiam.clustersetup.FailoverUtility.fetchServerConfiguration(FailoverUtility.java:31)
        at com.ca.eiam.clustersetup.FailoverConfigurator.startSecondaryFailoverConfiguration(FailoverConfigurator.java:907)
        at com.ca.eiam.clustersetup.FailoverConfigurator.performAction(FailoverConfigurator.java:380)
        at com.ca.eiam.clustersetup.FailoverConfigurator.commandPrompt(FailoverConfigurator.java:356)
        at com.ca.eiam.clustersetup.FailoverConfigurator.main(FailoverConfigurator.java:106)
ERROR - EE_SPONSORERROR iSponsor Error
[eemserver2.example.com]>

Environment

CA EEM on all supported platforms

Cause

• Communication problems between Primary and Secondary EEM servers:

 o   Network firewall blocking EEM ports.

 o   Name lookup (DNS) issues.

• Local Firewall (Windows firewall, Linux IPTABLES) on EEM hosts.

• EiamAdmin password incorrect/difference between EEM servers.

Resolution

Communications: 

• Ensure the port for CA iTechnology iGateway [TCP/5250] and CA Directory [TCP/509] are not blocked on the Network/System firewall. Enable the port for both inbound and outbound communication.
• Test the port communications using system tools (netstat, telnet, or nc)
• Ensure name resolution between EEM servers is good (nslookup using IP/FQDN/DNS names).
• Set the  EiamAdmin password to be the same on all EEM servers.

 Run a connection verification:

•       Attempt to connect the primary EEM server URL through a browser on the secondary host and vice versa.

The  below output is from a successful sync activity through eiam-clustersetup.jar after enabling port TCP/5250 on the Windows firewall between primary and secondary EEM servers:

C:\Program Files (x86)\CA\SC\EmbeddedEntitlementsManager\bin>java -jar eiam-clustersetup.jar -p eemserver1.example.com
Feb 2, 2016 2:24:44 PM IclUtil itechLibInit
INFO: iTechSDK initialized successfully
INFO  - EIAM_HOME [C:\Program Files (x86)\CA\SC\EmbeddedEntitlementsManager\]
INFO  - IGW_LOC [C:\Program Files (x86)\CA\SC\iTechnology\]
INFO  - DXHOME [C:\Program Files (x86)\CA\Directory\dxserver/]
INFO  - Hostname identified as [eemserver2.example.com]
EiamAdmin password :
INFO  - Checking server status
INFO  - igateway status      [started]
INFO  - dxserver status      [started]

Are you sure you want to continue? [Y/N]:y
[eemserver2.example.com]>sync
=======================================================
INFO  - Select current machine hostname
=======================================================
INFO  -    [1] eemserver1.example.com:509
INFO  -    [2] eemserver2.example.com:509
Select Hostname from [1 - 2] : 2
INFO  - Synchronization level
INFO  -    [1] [NEW] secondary node is being added first time
INFO  -    [2] [DELTA] secondary node is being synced to update configurations
Select Synchronization mode from [1 - 2] : 1
=======================================================
INFO  - Syncing with primary server
-------------------------------------------------------
Primary server                 Secondary server
-------------------------------------------------------
eemserver1.example.com    eemserver2.example.com
-------------------------------------------------------
INFO  - Re-configuring of server may lead to loss of data, it is advised to perform a backup of configuration and data store.

Are you sure you want to continue? [Y/N]:y
INFO  - Stopping dxserver service
INFO  - Stopping igateway service
INFO  - Configuring the knowledge group file
INFO  - ----------------------------------------------------
INFO  - The knowledge group file iTechPoz.dxg resides in the directory: C:\Program Files (x86)\CA\Directory\dxserver/\config\
knowledge\itechpoz.dxg
INFO  - Node was removed successfully
INFO  - Fetching configuration from eemserver1.example.com
INFO  - Adding self node [eemserver2.example.com]
INFO  - Generating:  : C:\Program Files (x86)\CA\SC\iTechnology\iAuthority.conf
INFO  - Generating:  : C:\Program Files (x86)\CA\SC\iTechnology\iControl.conf
INFO  - Generating:  : C:\Program Files (x86)\CA\SC\iTechnology\rootcert.cer
INFO  - Generating:  : C:\Program Files (x86)\CA\SC\iTechnology\rootcert.key
INFO  - Generating certificate for host : eemserver1.example.com
INFO  - Generating file : C:\Program Files (x86)\CA\Directory\dxserver/config\ssld\itechpoz-trusted.pem
INFO  - Adding failover node [eemserver1.example.com]
INFO  - Generating:  : C:\Program Files (x86)\CA\SC\iTechnology\iControl.conf
INFO  - Configuring DSA itechpoz for host eemserver2.example.com
INFO  - Configuring the itechpoz knowledge file
INFO  - ----------------------------------------------------
INFO  - The itechpoz knowledge file C:\Program Files (x86)\CA\Directory\dxserver/\config\knowledge\itechpoz.dxc.dxc  resides
in the directory: {2}
INFO  - Writing the itechpoz knowledge file
INFO  - Configuring DSA itechpoz-eemserver1.example.com for host eemserver1.example.com
INFO  - Configuring the itechpoz-eemserver1.example.com knowledge file
INFO  - ----------------------------------------------------
INFO  - The itechpoz-eemserver1.example.com knowledge file C:\Program Files (x86)\CA\Directory\dxserver/\config\knowledg
e\itechpoz-eemserver1.example.com.dxc.dxc  resides in the directory: {2}
INFO  - Writing the itechpoz-eemserver1.example.com knowledge file
INFO  - Configuring the knowledge group file
INFO  - ----------------------------------------------------
INFO  - The knowledge group file iTechPoz.dxg resides in the directory: C:\Program Files (x86)\CA\Directory\dxserver/\config\
knowledge\itechpoz.dxg
INFO  - Configuring the settings file
INFO  - ----------------------------------------------------
INFO  - C:\Program Files (x86)\CA\Directory\dxserver/\config\settings\itechpoz.dxc
INFO  - Clearing secondary server dsa-db
INFO  - Starting dxserver service
INFO  - Starting igateway service
INFO  - Run [status] to get server details.
[eemserver2.example.com]>exit

C:\Program Files (x86)\CA\SC\EmbeddedEntitlementsManager\bin>

Additional Information

EEM Failover Configuration: 

EEM Failover Configuration