New secondary Cluster Node install does not start and web UI not accessible
Article ID: 376565
Updated On:
Products
Issue/Introduction
After installing secondary cluster node, the given node is unable to start. Web UI is not accessible. Primary node is accessible.
Summary of errors seen during startup of secondary node
c2o.log:
20XX-XX-XX 14:53:17,373 ERROR [com.optinuity.c2o.jboss.StartUpInterceptor] [read Pool -- 98] Failed to decrypt data
java.lang.NullPointerException: null
20XX-XX-XX 14:53:17,398 ERROR [com.optinuity.c2o.server.ClusterConfigurationMgr] [read Pool -- 98] Error occured while initialising the System properties. Cannot proceed further
java.lang.Exception: Exception occured while decrypting a password
20XX-XX-XX 14:53:17,400 ERROR [com.optinuity.c2o.server.ClusterViewChangeListener] [read Pool -- 98] Got in exception when initializing the cluster
java.lang.Exception: java.lang.Exception: Exception occured while decrypting a password
20XX-XX-XX 14:53:17,403 ERROR [com.optinuity.c2o.server.ClusterViewChangeListener] [read Pool -- 98] Error 101: Cannot start ITPAM without cluster initialization. Server exiting.
server.log:
20XX-XX-XX 14:52:49,127 ERROR [org.apache.activemq.artemis.core.server] (Thread-2 (ActiveMQ-client-netty-threads)) AMQ224058: Stopping ClusterManager. As it failed to authenticate with the cluster: AMQ229099: Unable to authenticate cluster user: ACTIVEMQ.CLUSTER.ADMIN.USER
20XX-XX-XX 14:53:17,103 SEVERE [javax.enterprise.resource.webcontainer.jsf.flow] (MSC service thread 1-2) Unable to obtain CDI 1.1 utilities for Mojarra
20XX-XX-XX 14:53:17,114 SEVERE [javax.enterprise.resource.webcontainer.jsf.application.view] (MSC service thread 1-2) Unable to obtain CDI 1.1 utilities for Mojarra
20XX-XX-XX 14:53:17,367 ERROR [stderr] (ServerService Thread Pool -- 98) Bad Base64 input character at 8: 45(decimal)
Environment
Release 4.4 or higher
CA Process Automation
Cause
The primary node was configured to use 3rd party SSL certificate.
Best practise is to first install the secondary nodes and any other related PAM Servers, verify base functionality, then deploy 3rd party SSL certificates.
During the initial cluster setup, the OOB self-signed cert is used. After cluster is configured, the 3rd party certs should be deployed on all nodes
Resolution
Go into the standalone-full-ha and OasisConfig.properties files on node1 and set the SSL to use the default self-signed cert.
In the OasisConfig.properties (Located in C:\Program Files\CA\PAM\wildfly\standalone\.config) one should have these lines in place (uncommented):
itpam.web.keystorepath=C:/Program Files/CA/PAM/wildfly/standalone/.config/c2okeystore
itpam.web.keystore.password=[omitted]
itpam.web.keystorealias=ITPAM
In the standalone-full-ha.xml (Located in C:\Program Files\CA\PAM\wildfly\standalone\configuration), this line should be uncommented or rewritten to show as:
<keystore path="${itpam.web.keystorepath}" keystore-password="${KEYSTOREID}"/>
Recycle PAM.
Verify that node1 is now back to self signed SSL config
Access node2, and go into node1 webUI from node2
Verify node2 is already defined in the cluster setup in PAM
Run "Install Cluster Node for Domain Orchestrator" from node2.
Complete the install on node2 and verify that cluster setup is working
Perform the SSL config as follows:
- Stop node2 first.
- Stop node1 (will take time to stop as it is the master node)
- Node1: Setup SSL in the standalone-ha and Oasis.Configuration. Start services and verify node1 is up and reading the 3rd party SSL config
- Node2: Setup SSL in the standalone-ha and Oasis.Configuration. Start services and verify node2 is up and reading the 3rd party SSL config
Confirm that cluster is configured, and both nodes accept the 3rd party SSL cert.
Additional Information
There should be two references to SSL certificates in Oasis.Configuration (default self-signed SSL cert should not be commented out)
One can also check the folder PAM\wildfly\standalone\deployments and see if the file c2oear-snapshot.ear.deployed is missing and if it is missing, please copy the same file from a working node to here, end PAM service and then start PAM service.
Feedback
