We recently upgraded our system agents to R11.4.1 (R11.4 SP1) from R11.3.x. Customers started complaining about high memory usage with system agent on windows and sometimes jobs not responding as expected.
Release : 11.4
Component : CA Workload Automation System Agent
https://techdocs.broadcom.com/us/en/ca-enterprise-software/intelligent-automation/workload-automation-system-agent/11-5/release-notes.html#concept.dita_bd89790266bbc0a2aa13e33f3a0e96944a4bf911_id-.ReleaseNotesv11.5-ResolvedIssues
SYSAGT–338 Agent Process Consumes Excess Memory
The System Agent would consume an excess amount of memory after having been receiving scans from unexpected sources such as a vulnerability scanning tool.
Optional workaround for lesser versions
set the following in the agents agentparm.txt file and restart to limit the agents memory usage to 1gb.
oscomponent.jvm.x.options=-Xmx1g
NOTE - the memory issues typically happen as a result of port scanner from accessing the agent's port.