Applications Manager Launch Error and application lockups due to large out directory

Products

• CA Automic Applications Manager (AM)

Issue/Introduction

Applications Manager experiences severe performance degradation where users are disconnected and jobs fail to move from a Launched status to Running.

This typically culminates in RmiServer errors and Launch Errors at the 5-minute mark.

Symptoms:

• Users are kicked out of the application or it appears to lock up.
• Jobs stay in LAUNCHED status for several minutes before failing.
• Error AwE-9999 Internal error or AwE-5103 network socket error appears when attempting to open output.
• Log files report timeout SeqNo or Agent error : timeout during RMI operations.
• Standard output may show ERROR -999 ORA-20094: Jobid #### not in job queue.

Environment

Applications Manager 9.x.x
All supported OS platforms (Unix/Linux/Windows)

Cause

The $AW_HOME/out directory contains a large volume of files (typically exceeding 100,000).

The operating system experiences latency when reading or writing to a directory with this many entries, causing the Applications Manager RMI process to exceed its internal timeout limits.

Resolution

To restore performance, clear the output directory and verify retention settings:

1. Stop all Applications Manager processes.
2. Navigate to the $AW_HOME directory.
3. Rename the existing out directory to a backup name (e.g., out_old_####).
4. Create a new, empty out directory with the same permissions as the original.
5. Restart the Applications Manager processes.
6. Verify that jobs now move to a Running status without delay.

Some Jobs may abort and/or need to be restarted if Job completion is dependent on files that were previously in the out directory. Other Jobs may need moving of files from "old" out directory to new out directory before restarting aborted Job.

NOTES:

• The above suggestion is the "quick" fix to get Applications Manager up and running. Another option is to review and clean up the out directory of any files that are no longer needed, older the x days, etc.
• To check out directory for file count on Unix/Linux, the following command and be run when in the out directory:
  
  ls -1 | wc -l
  
  
• Check the so_retentn_days value for jobs to ensure output is not being kept longer then they need to be. To check current retention counts, run the following SQL query:
  
  SQL> select so_retentn_days, count(*) from so_job_table group by so_retentn_days order by 2 desc;