JCP queues flooded with messages

Products

• CA Automic Workload Automation - Automation Engine
• CA Automic One Automation

Issue/Introduction

During high load it can happen that the JCP queue gets flooded with messages (10.000+, up to 1 million)

In some occasions this may lead the Primary Work Process stopping to process even after the JCP is restarted which and lead to an outage.

Investigation

When the issue happens run the following query (Oracle only):

select count(*), utl_raw.cast_to_varchar2(dbms_lob.substr(MQCP_Msg, 200, 3)) as msg, mqcp_priority from mqXcpYYY
group by utl_raw.cast_to_varchar2(dbms_lob.substr(MQCP_Msg, 200, 3)), mqcp_priority order by 1 desc

where mqXcpYYY has to be adapted to the system:

• X is the current MQSET (1|2). The MQSET can be retrieved with "select ZDU_ACTIVE from UC_ZDU"
• YYY is the number of the JCP

If your system is suffering from the issue the query will return the following:

COUNT(*)	MSG	MQCP_PRIORITY
10.000+	<uc-env session=""> <kick system="UC4" client="client#" idnr="^0?">EH^ ? </kick></uc-env>	200
10.000+	<uc-env session=""> <kick system="UC4" client="other_client#" idnr="^0?">EH^ ? </kick> </uc-env>	200
...	...	200
30+	<uc-env session=""> <asyncmsg> <message date="yyyy-mm-ddThh:mm:ss" nr="xxxxxxxx" insert=" message"^ ?Ü/> </asyncmsg> </uc-env>	200
30+	<uc-env session=""> <asyncmsg> <message date="yyyy-mm-ddThh:mm:ss" nr="xxxxxxxx" insert=" message"^ ?Ü/> </asyncmsg> </uc-env>	200
...		200

 

Environment

Release : 12.2 / 12.3

Component : AUTOMATION ENGINE

Cause

This is a bug

Resolution

Workaround:

Restarting the JCP will empty its MQxCPyyy table.

If the MQxCPyy table contains many millions of entries it may be advantages to stop the JCP and truncate the corresponding MQxCPyyy table before restarting.

 

Solution:

Update to a fix version listed below or a newer version if available.

Fix version:

Component(s): Automation Engine

Automation.Engine 12.2.9 - Available

Automation.Engine 12.3.5 HF4 - Available

Automation.Engine 12.3.6 - Available