High Memory Utilization By System Agent
Article ID: 190473
Updated On:
Products
Issue/Introduction
Older agent versions would typically only average less than 100MB of memory usage. However, the newer agent versions are routinely using much larger amounts of memory.
Environment
Release : 11.4 and higher
Component : CA Workload Automation System Agent
Cause
The key difference between older and newer agent versions that account for the increased memory usage is the difference with the JRE version included with the agent. By default, there is nothing in the agent configuration that limits the maximum java heap allocation for the JRE. It relies on the java default for that. The older agents, which were using older JREs, would typically default the max heap to 256MB. Agents at version 11.4 and higher have a newer 64-bit JRE. Oracle changed the java default for heap allocation to where it now defaults the max to one-fourth of the total memory installed on the machine. Therefore, you will automatically see an increase in the max allocatable heap between older agent versions and agents that are version 11.4 or higher. This can be seen in the simple_health_monitor.log for the agent...
Old Agent
Maximum allocatable: 247 MB 512 KB 0 B
New Agent
Maximum allocatable: 3531 MB 0 KB 0 B
The next part of the equation is how much of the maximum heap has actually been allocated by the agent. That represents the actual memory usage. That is also visible in that same health monitor log...
Old Agent
Total allocated heap: 32 MB 568 KB 0 B
New Agent
Total allocated heap: 2741 MB 512 KB 0 B
Here we can see that your old agent has only allocated 32MB, where the new agent has allocated 2.7 GB. The agent only allocates memory when it needs it. However, the way java memory management works, once the memory has been allocated, it is not returned back to the O/S until the process that allocated it is restarted.
That leads to the final part of the equation, which is what causes the newer agent to allocate so much more memory than the older agent. We can see from the health monitor log that even though it has allocated 2.7GB, most all of it is always free...
Total allocated heap: 2741 MB 512 KB 0 B
Currently free heap: 2727 MB 839 KB 688 B
When the agent receives a message, the first thing it does is look at the first 4 bytes of the message to determine the full size of the message. If the full size of the message requires more than the max heap allocation, the agent will reject the message. If the full size of the message falls under the max, it will allocate the required amount of memory for the message, accept the remaining message, and process it. A message from AutoSys is limited to 32KB, so it would never cause that kind of allocation spike. What we have seen happening for other customers is there will be a message sent to the agent port that comes from some other application, like a port scanner, and the message is very large. In the old agent, messages like that would typically not fall within the 256MB max and get rejected with a memory error. However, since the newer agent has a much larger default max, the message size will fall under the max and cause the agent to allocate the necessary memory to receive it. Once it receives the rest of the message and tries to process it, it will produce an error in the receiver.log and discard it at that point if it is not a valid message from AutoSys or some other supported Broadcom scheduling product. While the memory used for the message is freed up, it remains allocated by the agent process and not returned to the O/S.
Resolution
oscomponent.jvm.x.options=-Xmx1g
This example is setting the max to 1GB. You can set it lower if you would like just by changing the value. For example, to set it to 256MB like the old agent...
oscomponent.jvm.x.options=-Xmx256m
256MB is typically plenty of memory for normal job processing. It is possible that more complex jobs that use some of the plugins like SAP might require more. However, if your old agent has been working ok at 256MB, and this new agent will be taking in similar workload, 256MB is probably enough for the new agent as well.
Additional Information
Other alternative would be to add following statement into <system_agent_install_dir>/cybAgent
ulimit -l 64
note: l = character l from limit
Feedback
