Some Agents are not connecting properly to AE with ERROR: duplicate key pk_mqsrv

Products

• CA Automic Workload Automation - Automation Engine

Issue/Introduction

Agent does not connect and stop generating anything else on the agent log until the agent was shut down by Service Manager or by killing the Agent process. While in this state, AWI shows it as inactive and will not run jobs.

20200709/101228.868 - U02000004 Connection to Server 'AESYSTEM#CP007(ID=3)' successfully created.
20200709/101228.868 - U02000354 CP Server 'AESYSTEM#CP00' reports ranking '1'.
20200709/101228.874 - U02000073 Connection to system 'AESYSTEM' via CP Server '' successfully established.
20200709/101228.879 - U02000066 Host information: Host name='agentname, IP address='127.0.0.1'

Trying to restart the CP on which the agent is connected to does not seem to resolve the issue and In the impacted CP log, we find these kind of errors:

U00003590 UCUDB - DB error: 'ERROR:  duplicate key value violates unique constraint "pk_mqsrv" DETAIL:  Key (mqsrv_name)=(AGENTNAME) already exists.'
U00003366 Connection to agent 'AGENTNAME' already exists (old connection '*CP003#00000262', new connection '*CP007#00000047').

The CP003 is now a JCP, instead of a normal CP.

Environment

Release : 12.3

Component : AUTOMATION ENGINE

Cause

Looking at the log of the CP to which the agent is connected to.

U00003412 Agent 'agentname' logged on (Client connection='2202').
U00003366 Connection to agent 'AGENTNAME' already exists (old connection '*CP003#00002202', new connection '*CP007#00000047').

Resolution

The following workaround can be used if you encounter this issue:

1. Stop JCP (on the example above is CP003)
2. Delete agent -> From AWI-> Client 0 -> Administrations-> Agents-> right click on the agent and choose Delete
3. Restart agent from Service Manager Dialog (or command line).
4. Start JCP

This mechanism of removing agents from MQSRV properly has been reviewed and fixed on Automation Engine v21.