Issue with EngineMgmtQueue MQ rising on master node

Products

• CA Process Automation Base

Issue/Introduction

There is an issue with EngineMgmtQueue MQ rising on master node, even though it's processing, this was after we failed over the master node.

Environment

Release : 4.3

Component : Process Automation

Cause

The count of the EngineMgmntQueue in the ActiveMQ msgs DB table was rising quickly.

Resolution

When running the DB query

select COUNT(*),CONTAINER from Node0ACTIVEMQ_MSGS group by CONTAINER 
select COUNT(*),CONTAINER from Node1ACTIVEMQ_MSGS group by CONTAINER
select COUNT(*),CONTAINER from Node2ACTIVEMQ_MSGS group by CONTAINER 
select COUNT(*),CONTAINER from Node3ACTIVEMQ_MSGS group by CONTAINER 

to view the counts of the response queues for the 4 orchestrators, it was noticed that although the response queue is fine (count changing up and down and processes are running), the EngineMgmntQueue (which is the rollup activity of moving completed records from the runtime to runtimearch table) count was continually increasing.

There was also an issue with one of the agent touchpoints not starting due to a problem with the service account that was used to start the service.

This was causing the error message "Error getting master node of Domain Orchestrator - Master node of Domain Orchestrator  not found" to be written out to the logs for all of the orchestrators.

The account issue for the agent was addressed, and the master node was failed back to the original master.

Once this was done, the count for the EngineMgmntQueue started dropping as expected.