EEM server is not starting, DSA is attempting to start after a long outage, perform a recovery procedure before starting

Products

Autosys Workload Automation

Issue/Introduction

The Autosys EEM process will not stay up.

The itechpoz_trace.log log show the following:
* [69] 20240215.083836.740 DSA_I2695 Multiwrite-DISP: Update from '<hostname>.<domainname>.com' applied
* [64] 20240516.062419.438 DSA_I1360 DSA received command to stop
* [64] 20240516.062419.438 DSA_I1240 DSA shutting down
* [64] 20240516.062419.530 DSA_I1360 DSA received command to stop
* [64] 20240516.062419.530 DSA_I1240 DSA shutting down
> [64] >>> address = tcp " '<hostname>.<domainname>.com'" port 509
? [64] 20240516.105803.547 ERROR : Syntax Error: Line 1 in /opt/CA/SharedComponents/CADirectory/dxserver/config/knowledge/ '<hostname>.<domainname>.com'.dxc near '509'
Invalid address '<hostname>.<domainname>.com'
* [64] 20240516.105803.564 DSA_E1280 Error in initialization files
* [64] 20240516.105803.565 DSA_I1240 DSA shutting down
* [64] 20240815.092229.584 DSA is attempting to start after a long outage, perform a recovery procedure before starting
* [64] 20240815.092229.586 DSA_I1240 DSA shutting down
* [64] 20240820.110747.259 DSA is attempting to start after a long outage, perform a recovery procedure before starting
* [64] 20240820.110747.260 DSA_I1240 DSA shutting down
* [64] 20240820.113849.849 DSA is attempting to start after a long outage, perform a recovery procedure before starting
* [64] 20240820.113849.855 DSA_I1240 DSA shutting down
* [64] 20240820.114520.579 DSA is attempting to start after a long outage, perform a recovery procedure before starting
* [64] 20240820.114520.579 DSA_I1240 DSA shutting down
* [64] 20240820.170009.906 DSA is attempting to start after a long outage, perform a recovery procedure before starting

Environment

Autosys 12.x
EEM 12.x

Cause

EEM setup in multi-write mode with another EEM and the secondary EEM is now down or not working properly.

Resolution

Here are the steps that should get your EEM servers back in sync...

Open a command prompt on both EEM servers.
In each command prompt, set the following variables...

set EIAM_HOME=C:\Program Files\CA\SharedComponents\EmbeddedEntitlementsManager
set JAVA_HOME=%EIAM_HOME%/jre
set PATH=%EIAM_HOME%/jre/bin;%PATH%

NOTE:
This assumes the SharedComponents directory is installed in the default location (C:\Program Files\CA\SharedComponents).
If your environment has it in an alternate path, specify the correct path for the EIAM_HOME variable.

Run the following on the primary EEM server...

cd %EIAM_HOME%/bin
java -jar eiam-clustersetup.jar
Type Y and press Enter.
Type 'remove' and press Enter.
Type the number corresponding to the secondary CA EEM Server that you want to remove and press Enter.
Type Y and press Enter.
Once it returns to the prompt, type 'list' and confirm that only the primary EEM server is listed.
Type 'resetprimary' and press Enter.
Press Enter to accept 509 for the DSA port
Select internal failover and press Enter.
Type Y and press Enter
Once it returns to the prompt, type 'exit' to return to the O/S prompt.

Leave the primary session open and move to the session on the secondary EEM server.
Run the following...

cd %EIAM_HOME%/bin
java -jar eiam-clustersetup.jar
Type Y and press Enter.
Type 'resetprimary' and press Enter.
Press Enter to accept 509 for the DSA port
Select internal failover and press Enter.
Type Y and press Enter
Once it returns to the prompt, type 'exit' to return to the O/S prompt.

Back over on the primary EEM server, run the following...

cd %EIAM_HOME%/bin
java -jar eiam-clustersetup.jar
Type Y and press Enter.
Type 'add' and press Enter.
Type the fully-qualified hostname of the secondary server, and press Enter.
Press Enter to accept 509 for the DSA port
Type Y and press Enter.
Once it returns to the prompt, type 'list' and confirm that both EEM servers are listed.
Type 'exit' to return to the O/S prompt

Back on the secondary EEM server, run the following...

cd %EIAM_HOME%/bin
java -jar eiam-clustersetup.jar -p <fullyqualifiedhostname_of_primaryserver>
Type the EiamAdmin password and press Enter.
Type Y and press Enter.
Type 'sync' and press Enter.
Type the number corresponding to the current hostname (secondary EEM server), and press Enter.
Type '1' and press Enter.
Type Y and press Enter.
Once it returns to the prompt, type 'exit' to return to the O/S prompt.

The EEM servers should now be in sync.