WP Started from Commandline Gets Breakpoint error

Products

• CA Automic Workload Automation - Automation Engine

Issue/Introduction

AE was upgraded from version 11.2 to 12.1.  Everything was working fine for a couple of days then AE went down. 

When the WP is restarted from the command line, it outputs a 'Breakpoint' message a few seconds later then the WP process dies.  However, the WP log shows it started up and is ready for run, but does not output a message that the WP ended abnormally.

Investigation
1. Get the count of the MQ1WP table:

SELECT COUNT(*) FROM MQ1WP;

2. The WP trace file shows this last SELECT statement before the WP dies: 

SELECT MQ1WP.*,ROWID FROM MQ1WP WHERE MQWP_SchedTime<=? ORDER BY MQWP_Priority, MQWP_SchedTime, MQWP_PK FOR UPDATE SKIP LOCKED

 

Actual: WP crashes when reading the message from the MQ1WP.
Expected:  Normally, the WP writes the message then crashes, but it does not do this. 

The trace file shows the bind parameter is:

MQWP_SchedTime = 2018.04.17 17:06:05.0

Example:

<Please see attached file for image>

3. The MQWP_SchedTime needs to be converted in order to run the SELECT statement against the DB:

SELECT MQ1WP.*,ROWID FROM MQ1WP WHERE MQWP_SchedTime<=(to_timestamp('17-04-2018 17:06:05', 'dd-mm-yyyy hh24:mi:ss')') ORDER BY MQWP_Priority, MQWP_SchedTime, MQWP_PK FOR UPDATE SKIP LOCKED

It should return one entry.

A few lines after in the trace file, the bind parameter for MQWP_PK can be found.

Run a SELECT from the MQ1WP table to get the message content:

SELECT * FROM MQ1WP WHERE MQWP_PK = 268859443 

Result: 805E3C75632D656E7620616464723D22503334422220636C69656E7456657273696F6E3D2231322E 
312E302B6275696C642E3933363422206E616D653D22464F524D532220726571756573743D226578 

Environment

AIX 7.1

Cause

An erroneous entry in the MQ1WP table, which translates to be an HTML message, coming from the AWI.
The ucsrv.ini file has this parameter set to:

exception=0

Resolution

Delete the entry from the MQ1WP table:

DELETE FROM MQ1WP WHERE MQWP_PK = 268859443

Once this entry is removed from the MQ1WP table, the WPs are able to start up.

Ensure this parameter in the ucsrv.ini file is set to:

exception=1

This will prevent wrong messages from entering the system.

Additional Information

Enable trace on the WP:

TCP/IP=2
Database=4

AIX OS core dump setting is enabled.