Unable to log on, Application Manager hangs on loading roles

Products

• CA Automic Applications Manager (AM)

Issue/Introduction

When attempting to log into the Applicaiton Manager the client hangs on "loading roles". 

The following error is eventually seen in the client window.

ErrorMsg: AwE-5009 RMI connection failure, check if RmiServer is running 
Details: null
java.rmi.RemoteException: Timeout SeqNo 11 Agent x.x.x.x:1102 Master Client service clientServices sessionID 41 Method getRolesData []; nested exception is:
Agent error : Timeout SeqNo 11 Agent x.x.x.x:1102 Master Client service clientServices sessionID 41 Method getRolesData [] : null

A review of the $AW_HOME/log/RmiServer<timestamp>.log shows the following when the process attempts to load the role data from the database using the procedure aw_web_api.awroles.

SocketManager$1: got request SeqNo 22 Agent 3.20.222.180:1102 Master Client service clientServices sessionID 43 Method getRolesData []
doRun 0 SeqNo 22 Agent x.x.x.x:1102 Master Client service clientServices sessionID 43 Method getRolesData []

From the logs the procedure aw_web_api.awroles does not get a return from the database or takes a really long time to return

 

Environment

OS Version: N/A

Cause

The aw_web_api.awroles procedure is being called, but then nothing is being returned back from the database or takes a long time to return. This may be due to issues with Oracle or the Network.

Resolution

A temporary workaround to see how long it takes for loading roles to finish loading is to increase the RMISocketTimeout from the default 300 seconds (5 minutes) to 900 seconds (15minutes) or more. 

1. In the directory where the RunClient.jar exist edit the client.properties file
2. Uncomment (remove #) the line first line "#localOptions=true"
3. Save/close the file.
4. In the directory where the RunClient.jar exist, in the <MASTER> sub-directory, edit the Options.properties file
5. Add the line "RMISocketTimeout=900" (If RMI error still occurs after 15 minutes of loading roles, further increase this number)
6. Save/close the file.
7. Open the RunClient.jar and attempt to log in.

While it is not efficient to have to wait 10 to 15 minutes to be able to log into the Applications Manager Client. This workaround is meant to be used a way to see how long it takes to finish loading roles as well as allow a User to log into the Java Client for further review.

Solution:

One of the reasons the "loading roles" behavior is seen is if there is a large number of substitution variables, Subvars, in the so_sub_vars table. This usually occurs if a Job condition is configured to create a dynamic Subvar based using the replacement value {jobid}, {chain_id}, or {topflow_id} as part of the Subvar name, AND is not being cleaned up. 

This causes a build up of the so_sub_vars table whose records are referenced and loaded during the "loading roles" process.

Deleting the extra unused dynamic subvars, either from the GUI or the database end should assist with improving or resolving the "loading roles" issue.

An example of a simple delete sql statement is below:

delete from so_sub_vars where so_var_name='<SUBVAR NAME>';

If needed, refer to your DBA for any additional sql statements.

If removing unused subvar records does not improve loading roles time, open a case with Broadcom Support for further review.

Support may request that an Oracle DBA or Network team be consulted to trace why the procedure (aw_web_api.awroles) does not get returned from the database or take a long time to return.

Additional Information

Workaround :
N/A