Jobs do not run or complete and are stuck in READY or PROCESSING state

Products

• DSERIES- SERVER
• CA Workload Automation DE - System Agent (dSeries)
• Workload Automation Agent
• CA Workload Automation AE - System Agent (AutoSys)

Issue/Introduction

The jobs stay in READY or PROCESSING state. This issue may affect only one or few UNIX and Linux agents. 

• UNIX agents installed on UFS or old JFS filesystems may experience this issue.
• Linux filesystems such as ext3 and ext2 may experience this issue.

Environment

UNIX Operating System such as Solaris and AIX on UFS and other non journaling filesystem

Linux on ext3 and ext2 filesystem.

Cause

Before running each job, the agent creates a directory to capture output in the spool. If the agent cannot make a directory for an application in spool directory, then the job will stay in READY state.

If a user is unable to create a directory (not touch a file) manually, and the OS gives following error:  

mkdir: Failed to make directory "directory_name"; Too many links.

Then, the issue with the OS filesystem.

Resolution

The issue is with the limit of subdirectories within a directory. The UFS and older Linux filesystem only allow 32,726 (2^15) subdirectories. To create new directories, older directories will have to be removed or cleared out.

Note:  User may run the following command in the spool directory to get the directory count. For example;

cd /<agent_install_dir>/spool/MANAGER_INSTANCE/MAIN

ls -d */ | wc -l

User may also archive the spool directory by renaming it to something like spool.old and restart the agent. The agent will create a new spool directory upon restart. However, it is highly recommended to periodically clean the spool. It can also be done by the agent by enabling the following parameters in the agentparm.txt (agent restart is required)

runnerplugin.spool.clean.enable=true 
runnerplugin.spool.expire=7D 
runnerplugin.spool.sleep=24H  

If above parameters are enabled already, then reduce the 'runnerplugin.spool.expire' value to keep only few days of spool files.

Additional Information

For more information on the above parameters, please consult Workload Automation Agent documentation here.