[Unresolved] Linux Agent maxes CPU until restarted

Products

• CA Automic One Automation

Issue/Introduction

LINUX ONE Automation agent maxes CPU on app servers

This occurs about one or two times per month.

Restarting the agent frees up the CPU usage

Environment

Release : 12.3

Component : AUTOMATION ENGINE

Resolution

Root cause was not found as reproduction with correct tracing was unable to be done.  If this issue occurs, please gather it with the information below before opening a case with support:

strace:

1. run a ps-ef|grep ucxjlx6 command to get the pid for the agent and note the pid
2. run the command:
   strace -tt -fp [pid from step 1] -o [output file]
   where [pid from step1] is replaced by the pid for the process from step 1 and [output file] is a good output file to push out the monitoring from the strace command
3. Let this run about 5 minutes and then cancel out.  

For example:
Running a ps -ef|grep ucxjlx6 command from the agent's bin directory returns the following result:
root      91323      1  0 Oct11 ?        00:03:20 /opt/uc4/1236agent/bin/ucxjlx6

The PID is 91323 - then run:
strace -tt -fp 91323 -o /opt/strace_out.txt

Let this run for about 5 minutes and then Ctrl+C out of the command.  That file will be neded.  So full instructions at this point are:

1. Note how long it takes for the CPU usage to increase to the level that the agent needs to be restarted.
2. After a restart of the agent, find the agent pid and run a strace command like this:
   
   strace -tt -fp [pid from step 1] -o [output file]
3. Let that run for about 5 minutes
4. About half way to the time that CPU usage will be a max level, report on CPU usage for the agent and all child processes using the same method as from step 2 above
5. When it's about to hit the max usage, please turn on an agent trace:
   
   Go into the Administration perspective, go to Agents
   Right-click on the agent and choose Advanced Options
   Change TCP/IP from 0 to 9 and hit Apply
   Let this go about 10 minutes
   Go back to Advanced Options on the agent and change TCP/IP back to 0
   
   
6. Right-click on the agent in the client where it used to run things, and choose "Monitoring" and then "Show in Process Monitoring".  
7. From Process Monitoring, click on "Export table" in the upper right
8. Report on CPU usage again for the agent and all child processes using the same method as from step 2 above
9. Then please:

Send Support the results from 1, 2, 3, and 8 above
Send Support the agent log file ending in 00.txt
Send Support the agent trace file ended in 00.txt
Send Support the file that was created from the "Export table" in step 6 above