U02003043 Invalid 'fork' call. Error code: ('12' - 'Cannot allocate memory')

Products

• CA Automic Workload Automation - Automation Engine
• CA Automic One Automation
• CA Automic Operations Manager
• CA Automic Oracle

Issue/Introduction

After some time, an Unix Agent may have jobs begin to fail with:

U02003043 Invalid 'fork' call. Error code: ('12' - 'Cannot allocate memory')

Cause

This is due to the inability of the Agent on an OS level to complete a fork for the execution of a JOBS.

Environment

Release : 12.3

Component : AUTOMATION ENGINE

Resolution

There is no single resolution. To further troubleshoot:

Provide the correct "ulimit -a" and "free -m" and "swapon -s" from the machine where the agent is running, in addition to the implemented limits in /etc/security and the pam config, and the values for vm.overcommit_memory and vm.overcommit_ratio.
To check what is going on on the system, output of 'ps aux --sort -rss'
Check the memory consumption of the agent with a 'pmap -x $PID' for all Agent processes.
Please provide matching agent logfiles and tracefiles with tcp/ip=9, job_debug=9
A strace of the agent with 'strace -tt -ff -o strace.out -p <Agent PID here>' during occurrence of the "Invalid 'fork' call. Error code: ('12' - 'Cannot allocate memory')’"
A strace of the listener with 'strace -tt -ff -o strace.out -p <listener PID here>' during occurrence of the "Invalid 'fork' call. Error code: ('12' - 'Cannot allocate memory')’"

To workaround this issue, restart the Agent.