Jobs do not run or complete and are stuck in READY or PROCESSING state.

Products

• DSERIES- SERVER
• CA Workload Automation DE - System Agent (dSeries)

Issue/Introduction

The jobs stay in READY or PROCESSING state. This issue may affect only one or few UNIX and Linux agents. 

UNIX agents installed on UFS or old JFS filesystems may experience this issue. Some older Linux filesystems such as ext3 and ext2 may experience this issue.

Cause

Before running each job, the agent creates a directory to capture output in the spool. If the agent cannot make a directory for an application in spool directory, then the job will stay in READY state in DE. If a user is unable to create a directory (not touch a file) manually, and the OS gives following error:  

mkdir: Failed to make directory "directory_name"; Too many links.

 Then the issue with the OS filesystem.

Environment

UNIX OS such as Solaris and AIX on UFS and other non journaling filesystem.Linux on ext3 and ext2 filesystem.

Resolution

The issue is with the limit of subdirectories within a directory. The UFS and older Linux filesystem only allow 32,726 (2^15) subdirectories. To create new directories, older directories will have to be removed or cleared out.

 

Note:  User may run the following command in the spool directory to get the directory count:

cd /<agent_install_dir>/spool/MANAGER_INSTANCE/MAIN

ls -d */ | wc -l

User may also archive the spool directory by renaming it to something like spool.old and restart the agent. The agent will create will create a new spool directory upon restart. However, it is highly recommended to periodically clean the spool. It can also be done by the agent by enabling the following parameter in the agentparm.txt (agent restart is required)

runnerplugin.spool.clean.enable=true 
runnerplugin.spool.expire=10D 
runnerplugin.spool.sleep=24H  

For more information on the above parameters, please consult CA Workload Automation Agent 11.3 documentation here.