Unable to log in to EEM - ISE_BACKENDDOWN backend is down

Products

Process Automation Manager CA Service Management - Asset Portfolio Management CA Service Desk Manager CA Service Catalog Autosys Workload Automation

Issue/Introduction

In EEM, there are a few reasons why the underlying DSA database will get out of sync between multiple EEM servers in a failover environment. This causes issues with logging in and/or authorizing with the attached/registered applications.

This document provides a step by step approach to alleviate the issue (assumes a cluster of 2 EEM servers only for simplified use case)

Environment

Embedded Entitlements Manager

Cause

Potentially an IP address change, a network issue where the servers cannot communicate, the Directory database may fill up and no longer be able to update.
If a message below is seen in the itechpoz_trace.log file under $DXHOME (CA Directory Home folder), this might require further manual intervention.

DSA_E2735 Multiwrite-DISP: Unable to synchronize with peer 'itechpoz-Failover-hostname’

Resolution

The first step will be to disconnect the two EEM servers. This will cause an outage, please plan accordingly and always do this in a test environment first. This is outlined in the EEM documentation as well.
It is required that the EiamAdmin password be identical on all EEM servers to be in the cluster.
The eiam-clustersetup utility is to be used to remove the secondary CA EEM Servers from the primary CA EEM Servers. After the secondary EEM Server is removed, a reset on the primary EEM Server needs to be performed.
On the Primary CA EEM Server:

#change directory to EIAM_HOME/bin
# set JAVA in environment

cd $EIAM_HOME/bin
JAVA_HOME=$EIAM_HOME/jre; export JAVA_HOME
PATH=$JAVA_HOME/bin:$PATH; export PATH

#Execute the following command:
java -jar eiam-clustersetup.jar

#A confirmation message appears.
Type Y and press Enter.

#Execute the following command:
remove

#The following message appears. Select hostname
Type the number corresponding to the secondary CA EEM Server that you want to remove and press Enter.

#A confirmation message appears.
Type Y and press Enter.

#Continue to repeat the above steps for all the names of the secondary EEM Servers.

# Once the above is completed for all the names of secondary servers, execute the following command
list

#Only the primary EEM Server name should be listed there. Any secondary EEM Servers should not show up here anymore.

#To reset the primary CA EEM Server, execute the following command:
resetprimary

#The following message appears:
Enter DSA Port [default = 509]

#If necessary, update the default DSA port number and press Enter.

# Then the following message appears:
Specify high-availability mode

#Select a high-availability mode and press Enter.
#A confirmation message appears.
Type Y and press Enter.

#The primary EEM Server is now reset.

# To close the eiam-clustersetup utility, execute the following command:
exit

The secondary CA EEM Servers are removed from the primary CA EEM Server and the primary CA EEM Server is reset.
On the Secondary EEM server:

#change directory to EIAM_HOME/bin
# set JAVA in environment

cd $EIAM_HOME/bin
JAVA_HOME=$EIAM_HOME/jre; export JAVA_HOME
PATH=$JAVA_HOME/bin:$PATH; export PATH

#Execute the following command:
java -jar eiam-clustersetup.jar
#Type Y and press Enter.

#Execute the following command:
resetprimary

#The following message appears:
Enter DSA Port [default = 509]

#If necessary, update the default DSA port number and press Enter.

# Then the following message appears:
Specify high-availability mode

#Select a high-availability mode and press Enter.
#A confirmation message appears.
Type Y and press Enter.

#The secondary EEM Server is now reset.

# To close the eiam-clustersetup utility, execute the following command:
exit

ON ALL EEM Servers - Stop all EEM services on all the servers (Stop the iGateway service first and then the CA Directory iTechPoz)
On the Primary EEM server:

#change directory to dxserver/data/itechpoz
cd /opt/CA/Directory/dxserver/data/itechpoz

#There are two files here
itechpoz.db
itechpoz.tx

#Make a copy of the itechpoz.db file and move this copy to the secondary EEM server to a temp folder (example: /tmp)
On the Secondary EEM server ,

#change directory to dxserver/data/itechpoz
cd /opt/CA/Directory/dxserver/data/itechpoz

#Delete the itechpoz.tx file
rm -f itechpoz.tx

#Rename the itechpoz.db file to itechpoz.db.orig - NOTE: If it is Windows, esure to show file extensions enabled, a wrong file maybe renamed.
mv itechpoz.db itechpoz.db.orig

#Copy the /tmp/itechpoz.db file (that was copied from the Primary server) to /opt/CA/Directory/dxserver/data/itechpoz
cp -p /tmp/itechpoz.db /opt/CA/Directory/dxserver/data/itechpoz

#Now there should be two files in this folder:
itechpoz.db
itechpoz.db.orig
On the Primary EEM server:

# Start the EEM primary services. The Directory service must be started before iGateway service.

# Then attempt to login as EiamAdmin into the Primary EEM UI using the Global Application.

# If the above does not work, stop here and engage Broadcom support via a case for further assistance. (A clean install may be required for the EEM primary server)
On the Secondary EEM server:

# If the EiamAdmin login works on the Primary EEM UI, start the dxserver and iGateway services on the secondary server.

# Attempt to login as EiamAdmin into the Secondary EEM UI using the Global Application. If you cannot log in, again stop here and call support. A clean install of the secondary may be required.

# If both the above worked fine, next step is to establish HA back again
On the Primary EEM server:

#change directory to EIAM_HOME/bin
# set JAVA in environment

cd $EIAM_HOME/bin
JAVA_HOME=$EIAM_HOME/jre; export JAVA_HOME
PATH=$JAVA_HOME/bin:$PATH; export PATH

#Execute the following command:
java -jar eiam-clustersetup.jar
#Type Y and press Enter.

#Execute the following command:
add

#Type the fully-qualified hostname of the secondary server, and press Enter.

#The message "Enter DSA Port [default=509]" appears. Do one of the following steps:
If you want to accept the default port, press Enter.
If you want to enter a different DSA port, type the port number and press Enter.
Type Y and press Enter.

#The failover tool starts validating the entered details. After the validation, the secondary server is added to the primary server.

#The next step is to synchronize the configuration of the primary server with the secondary servers in a failover setup. After the synchronization, the configuration of each secondary server in the failover setup is overwritten with the configuration of the primary server.
On the Secondary EEM server:

#change directory to EIAM_HOME/bin
# set JAVA in environment

cd $EIAM_HOME/bin
JAVA_HOME=$EIAM_HOME/jre; export JAVA_HOME
PATH=$JAVA_HOME/bin:$PATH; export PATH

#Execute the following command:
java -jar eiam-clustersetup.jar -p <fullyqualifiedhostname_of_primaryserver>

# The installation paths of EEM, Directory, and iTechnology iGateway and the status of the failover tool on the secondary server details are displayed.

#The command-line interpreter changes from $ to the hostname of the secondary server.

#Type the EiamAdmin password and press Enter.
#Type Y and press Enter.

#Execute the following command:
sync

#Type the number corresponding to the current (NOTE: this will be the Secondary hostname, NOT the primary) hostname, and press Enter.

#As you are synchronizing the secondary server with the primary server for the first time, type 1 and press Enter.

#Type Y and press Enter.

#The failover tool starts validating the synchronization process. After the validation, the secondary server is synchronized with the primary server.

NOTE: If the sync command does not work and gives any errors during authentication of eiamadmin earlier, raise a case with Broadcom Support for further help

Once the above is successful, log in to both EEM UIs as EiamAdmin to the Global application to verify that they function.
If an external (LDAP) directory store was configured on the Primary server, the same needs to be configured manually on the secondary server. This userstore configuration does not replicate between the servers, meaning, changes have to be done manually to all EEM servers in the cluster.

Additional Information

To verify that replication is working as expected, log in to the primary EEM server as EiamAdmin to the Global application.
Select Manage Access Policies
Create a test policy.
Remember where you created this.
Once the test policy is saved, log out.
Log in to the secondary EEM server as EiamAdmin to the Global application.
Select Manage Access Policies
The test policy created earlier should show up.
From here, delete the test policy and save.
Log in once again to the primary EEM server as EiamAdmin to the Global application.
Select Manage Access Policies
The the test policy should not show up anymore
If all of the above is successful as outlined, the replication is fully functional.

NOTE: The key to all of this is that the EiamAdmin passwords are identical on all EEM servers. If they are not, none of this will work.