How to monitor and alert on AutoSys WAAE service status

Products

• Autosys Workload Automation

Issue/Introduction

The AutoSys Workload Automation AE (WAAE) Scheduler or other services (Application Server, Web Server) stop unexpectedly without warning. There is a requirement to monitor these services and receive alerts when a service is down.

SYMPTOMS

• The event_demon (Scheduler), as_server (Application Server), or waae_webserver processes stop.
• Review of the standard WAAE logs shows no obvious errors, deadlocks, or failures at the time of the service stop.
• Standard log paths checked:
  • $AUTOUSER/out/waae_webservices_access_log.<date>.log
  • $AUTOUSER/out/event_demon.$AUTOSERV
  • $AUTOUSER/out/as_server.$AUTOSERV
  • Agent logs: transmitter.log, receiver.log

Environment

• Product: AutoSys Workload Automation AE
• Component: Scheduler, Application Server, Web Server
• Platform: Linux / UNIX

Cause

AutoSys Workload Automation AE does not include a native internal email alerting system for its own service process status.
When services stop "without warning" and no application-level errors are logged, it is often due to external factors such as OS-level process termination (e.g., OOM kills), database connectivity drops, or hardware interruptions.

Resolution

To monitor and alert on AutoSys service status, use one or more of the following enterprise monitoring strategies:

1. Use the unisrvcntr Utility

The unisrvcntr status command provides a consolidated view of all WAAE services and their PIDs.

• Action: Create a shell script that periodically runs unisrvcntr status and parses the output for any component not in a "running" state.
• Alerting: Configure the script to send an email or SNMP trap if a service is missing.

2. Operating System Level Monitoring

Monitor the specific processes at the OS level (e.g., event_demon, as_server, waae_webserver).

• Action: Use enterprise monitoring tools (such as DX UIM, Zabbix, or Nagios) to monitor the process table.
• Alerting: Set up alerts to trigger if these processes disappear from the process list.

3. SNMP Traps

AutoSys can be configured to send SNMP traps to an enterprise monitoring tool when services change state.

• Action: Ensure SNMP is configured in the AutoSys instance. This is typically set during installation but can be enabled or adjusted in the configuration files post-install.

4. Review System Logs

If services stop with no application errors, check the OS system logs for external termination signals.

• Action: Check /var/log/messages, /var/log/syslog, or run dmesg to look for "Out of Memory" (OOM) kills or segmentation faults related to the WAAE processes.

5. Database Connectivity Check

The Scheduler will shut down immediately if it loses its connection to the database.

• Action: Monitor the database listener and network connectivity between the Scheduler server and the database server.
• Log Check: Double-check the event_demon.$AUTOSERV log for any "deadlock" or "database synchronization" errors that might precede a shutdown.