Message Queue is showing "too many open files"

Products

• CA Privileged Access Manager - Server Control (PAMSC)
• CA Privileged Identity Management Endpoint (PIM)

Issue/Introduction

CA Privileged Identity Manager Message Queue is showing this error in its log file

# tail /opt/CA/AccessControlDistServer/ACMQ/acrptmq.log
...
2016-03-10 11:10:24.303 ERROR: Accept() failed: too many open files. Please check per-process and system-wide limits on the number of open files.
...

Hence e.g. on the DMS we see a high number of pending messages or other issues related to malfunctioning Message Queue.

Environment

Release: R 12.9-Privileged Identity Manager - Linux
Component: CA Privileged Identity Manager

Cause

System wide user and process limits are not honoured by the tibemsd process (which is the main process of the CA Privileged Identity Manager Message Queue).

This can be verified by running these commands in a root shell on the affected box:

# cat /proc/`pidof tibemsd`/limits | grep 'Max open files'
Max open files            1024                 4096                 files

# ls -l /proc/`pidof tibemsd`/fd | wc -l
1025

As it can be seen tibemsd is trying to allocate more file descriptors (here 1025) than its set limits (here 1024)

Resolution

Below is a workaround to fix the problem.

Please add either one of the two lines to the CA Privileged Identity Manager Message Queue startup script as shown below:

# vi /etc/init.d/ca-acrptmq
...
start() {
...
        # echo -n "Max open files=20000:20000" > /proc/`pidof tibemsd`/limits
        echo -n "Max open files=20000:20000" > /proc/`getmqpid`/limits
        return $RETVAL
}
...

Restart the CA Privileged Identity Manager Message Queue or even reboot the box and verify the error is disappeared and no pending messages are happening.

Additional Information

Please see the Linux documentation for how to configure system wide and process specific limits.