Production down – CP ends for no reason at all.

Products

• CA Automic Workload Automation - Automation Engine

Issue/Introduction

Production down – CP ends for no reason at all.

Environment

Release: AOATAM99000-9.0-Automic-One Automation Tools-Application Manager
Component:

Resolution

CP ends for no reason at all. 

In the CP logs there are following information:
20130319/121920.637 - U0003406 Client-Verbindung '3190(1016)' von '10.206.24.51:1957' hat sich am Server angemeldet.
20130319/121920.637 - U0003413 Socket-Aufruf 'recv(3190)' ergab Fehlercode '11'.
20130319/121920.637 -          Resource temporarily unavailable

As you can see there are 1016 active connections on the process.
On the next step I controlled the ulimit values:
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
 file size               (blocks, -f) unlimited
 pending signals                 (-i) 223505
 max locked memory       (kbytes, -l) 64
 max memory size         (kbytes, -m) 24325552
 open files                      (-n) 1024
 pipe size            (512 bytes, -p) 8
 POSIX message queues     (bytes, -q) 819200
 real-time priority              (-r) 0
 stack size              (kbytes, -s) 8192
 cpu time               (seconds, -t) unlimited
 max user processes              (-u) 223505
 virtual memory          (kbytes, -v) 39671840
 file locks                      (-x) unlimited


Here you can see that the "open files" was limited to 1024.

After changing the "open files" to unlimited, the system works correctly.
Workaround in this situation:
Start additional CP to distribute the connections and decrease the count of connections for each process.