'Too many open files' error=24 on Remote Agents

Products

• CA Automic Applications Manager (AM)

Issue/Introduction

Error=24 'Too many open files' error on Remote Agents.

Error Message Details:

ErrorMsg: AwE-9999 Internal error 

Details: Cannot run program "sh" (in directory "<path>"): error=24, Too many open files

Environment

CA Automic Applications Manager

Version 9.2 and higher

Cause

Files are left open when running an Applications Manager remote agent job on some operating systems.  This is an OS related issue and indicates that the amount of open files for a user exceeds the number in the setting. All jobs on a remote Agent are going into a LAUNCH ERROR status. The log files show the above error.

Resolution

For Solaris Only:

Add the following line to the [default] section of the $AW_HOME/site/awenv.ini file of the Agent:

CloseProcessStreams=<number of seconds>

The value will be the number of seconds between check intervals, a recommended value is 300. For every job launched, the process object will be added to a list and check to make sure the process is still running in an interval of <CloseProcessStreams> seconds. If the module is no longer running, the stdout, stderr and stdin streams will be closed.

For All Other Operating Systems:

You may need to refer to your OS admin to increase the number of open files and open file descriptors at the OS level.  Below you will find some basic settings.  However, if these limits are not enough, they may need to be increased further in your environment. 

SYSTEM Wide Settings:

To see the settings for maximum open files for the OS level, use the following command:

     cat /proc/sys/fs/file-max

This should be a value from 36000 to 50000 or more.  To increase the system wide maximum open files, as root edit the /etc/sysctl.conf and add the following to the end of the file.

     fs.file-max = 49500

NOTE:  The above example will increase the maximum number of files to 49,500 on your currently running system and will persist after rebooting.

Then issue the following command to activate this change on your system:

     sysctl -p

USER Settings:

To see the settings for a user, as root issue the following commands:

 

 

     su - <user>
     ulimit -n

The default setting for this is usually 1024.  If more is needed for a specific user than as root modify it in the /etc/security/limits.conf file:

     @user -nofile 2048

Will set the maximum open files for the specific "user" to 2048 files.

To do a system wide increase for all users as root edit the /etc/security/limits.conf file and add the following:

     * - nofile 2048

This sets the maximum open files for ALL users to 2048 files.  These settings will require a reboot to become active.