ITPAM orchestrator cluster node is not starting up

Products

• CA Process Automation Base

Issue/Introduction

After removing and rebuilding one node of a cluster, the other nodes are not starting up.

Issue may also happen after a Windows server update while retaining the existing orchestrator nodes

Environment

CA PAM 4.3 SP5 on Windows 2012

 

Cause

For scenario where a node was removed, reviewed c2o.log and found this entry:

20:06:08,476 ERROR [org.jgroups.protocols.TCPPING] [ main] failed creating initial list of hosts 

java.net.UnknownHostException: SERVER.EXAMPLE.COM

Confirmed that this is the server that no longer existed. 

 

Other scenarios may include c2o.log log entries such as this:

13:34:15,649 ERROR [com.optinuity.c2o.server.ServerController] [ Initialization] Failed to obtain unique name for local cluster node
javax.naming.NameNotFoundException: HAPartition not bound
 
13:37:17,686 ERROR [com.optinuity.c2o.c2oserver.ServerManager] [ Initialization] Server Manager Failed to boot - timeout waiting for database structure adjustment

16:26:16,054 ERROR [com.optinuity.c2o.server.ServerController] [ Initialization] Failed to obtain unique name for local cluster node
javax.naming.NameNotFoundException: HAPartition not bound

16:26:21,179 WARN  [com.optinuity.c2o.util.SingletonMBeanLookupUtil] [ Initialization] Failed to create server connection to singleton MBean: c2o:service=OasisServerController in partition: eab13682-d8ed-444e-be0b-3bf05c5ed792
javax.naming.CommunicationException: Receive timed out

Resolution

Perform the following to clear the existing cluster nodes configuration

• Stop CA PAM services on all nodes that participate in a given cluster. 
• Backup the backend CA PAM Runtime database, at the minimum, the "properties" table should be backed up.
• Delete all records from the "properties" table from the CA PAM Runtime database. 
  
  • CA PAM will update the database when the Orchestrators are started. 
• Move the following folders out of the PAM server\c2o directory: audit, data, scripts, tmp, temp, work, wrappers, and .recovery
  • Please move these folders to another location on the server.  
• Start CA PAM services on the nodes that will remain active. 

 

 

Additional Information

Uninstalling CA PAM on the Orchestrator that will not participate in the Cluster will not clean up the entries in the database. It only cleans up the files, folders, and services on the server. The steps above are the steps that should be followed for making sure that node is cleaned up from the database.