Troubleshooting memory leak issues with autosys application server / scheduler

Products

• Autosys Workload Automation

Issue/Introduction

This document provides ways to troubleshoot memory leak issues with autosys as_server and event_demon

Resolution

A process growing in RSS isn't automatically a leak.

For example, AutoSys scheduler caches job/calendar data in memory, and RSS can plateau. A leak means unbounded, monotonic growth with no plateau, ideally correlated with elapsed time or event volume, not just "it uses more RAM than I expected." 

# sample RSS over time — run this for a few hours minimum, ideally 24h+
while true; do
ps -o pid,rss,vsz,etimes,cmd -p $(pgrep -f event_demon) >> /tmp/event_demon_mem.log
date >> /tmp/event_demon_mem.log
sleep 300
done

## Plot rss vs etimes. Linear/unbounded growth with no ceiling = real leak signal. Growth that flattens after startup/cache-warm = probably not.

Narrow down where

• /proc/<pid>/status — check VmRSS, VmData, VmSwap over the same sampling window. If VmData grows but VmRSS doesn't, it may be virtual/reserved, not resident — much less urgent.
• pmap -x <pid> snapshots over time (e.g. hourly) — diff them to see which memory regions are growing (heap, a specific shared library's mapping, anon mmap segments). If growth is concentrated in a library's mapped region rather than the main heap, that points at a third-party dependency (this toolkit's local KB has prior cases of this — see below) rather than AutoSys's own code.
• Correlate with activity — does RSS jump per job-run, per DB reconnect, per event processed?

 

To really debug core memory leak type of issues, valgrind could be used.  Here is an example:

• Install valgrind first
  yum install valgrind (or dnf install valgrind)
 
• Terminal #1 stop scheduler and start it manually via valgrind:
unisrvcntr stop waae_sched.$AUTOSERV
valgrind --trace-children=yes --log-file=/tmp/vg.out --num-callers=50 --error-limit=no --leak-check=full --show-reachable=yes event_demon -A $AUTOSERV 
   
• Leave this running via the terminal#1 until below step is achieved
   
• Via a second terminal, continue monitoring for minutes/hour until you notice memory leak
   
• Then via terminal#2, stop scheduler:
unisrvcntr stop waae_sched.$AUTOSERV
   
• Gather scheduler logs + /tmp/vg.out file for review

 

 

Similar approach can be taken for as_server (waae_server.$AUTOSERV) also.

 

Provide all the above gathered data to Broadcom Support team for analysis.

Additional Information

To speak with a customer representative or a Support Engineer see Contact Support. Scroll to the bottom of the page and click on the respective region.