Processes stuck in Waiting or Running status without moving forward

Products

• CA Process Automation Base

Issue/Introduction

Existent processes are all in Waiting state and new processes get stuck on Running state too.

Environment

Process Automation - All supported versions

Cause

c2o.log is filled with messages like this:

INFO  [com.optinuity.c2o.transport.Resolver] [igUpdatesThread] Transport properties [TransportID=agentNode, Hostname=host.domain.LOCAL, IPAddress=xxx.xxx.xxx.xxx, Port=7003, IsSecure=false] of node 5a0928de-1ddb-4eb3-a8ee-a213496db353 not reachable
WARN  [com.optinuity.c2o.c2oserver.ServerManager] [igUpdatesThread] Error sending Domain update message to 5a0928de-1ddb-4eb3-a8ee-a213496db353

INFO  [com.optinuity.c2o.transport.Resolver] [igUpdatesThread] Transport properties [TransportID=agentNode, Hostname=host.domain.local, IPAddress=xxx.xxxx.xxx.xxx, Port=7003, IsSecure=false] of node ffbd8b60-8c5a-445e-9811-ca02c9456d17 not reachable
WARN  [com.optinuity.c2o.c2oserver.ServerManager] [igUpdatesThread] Error sending Domain update message to ffbd8b60-8c5a-445e-9811-ca02c9456d17
com.optinuity.c2o.util.C2OException: Unidentified UUID while resolving node

INFO  [com.optinuity.c2o.transport.Resolver] [_autoTPRecovery] Transport properties [TransportID=agentNode, Hostname=host.DDOMAIN.LOCAL, IPAddress=xxx.xxx.xxx.xxx, Port=7003, IsSecure=false] of node 0d14b475-934f-462c-98d4-f1cf25848e84 not reachable
ERROR [com.optinuity.c2o.c2oserver.ServerManager] [_autoTPRecovery] Agent with id : 0d14b475-934f-462c-98d4-f1cf25848e84 is not reachable
com.optinuity.c2o.util.C2OException: Unidentified UUID while resolving node

INFO  [com.optinuity.c2o.transport.Resolver] [_autoTPRecovery] Transport properties [TransportID=agentNode, Hostname=host.domain.local.net, IPAddress=xxx.xxx.xxx.xxx, Port=7003, IsSecure=false] of node 43173236-da8e-4f73-bd6e-2a26a809ef8b not reachable
ERROR [com.optinuity.c2o.c2oserver.ServerManager] [_autoTPRecovery] Agent with id : 43173236-da8e-4f73-bd6e-2a26a809ef8b is not reachable
com.optinuity.c2o.util.C2OException: Unidentified UUID while resolving node

All those Agents exist, but they are not active:

Resolution

The Orchestrator is trying to reach all those Agent several times in a row, which causes an overflow on the system, that's why the Orchestrator is no able to process all the Running/Waiting instances.

The way to solve this is to delete all inactive Agents/Touchpoints as per:

Delete a Touchpoint

Delete an Agent