Process Automation - ActiveMQ not starting after an abnormal stop of server

Products

• CA Service Management - Service Desk Manager
• Process Automation Manager
• CA Process Automation Base

Issue/Introduction

Sometimes, when an abnormal stop happens on the servers (electricity issues, VM reboot, etc.) and some installed features may be impacted.

If after an abnormal stop PAM's login page is not shown (even though the Orchestrator service starts), the problem is caused by ActiveMQ module not starting correctly.

In c2o.log the following can be seen:

INFO  [com.optinuity.c2o.c2oserver.ServerManager] [ Initialization] Starting ActiveMQ: 
INFO  [com.optinuity.c2o.c2oserver.ServerManager] [ Initialization] Checking if ActiveMQ has been started...
INFO  [com.optinuity.c2o.c2oserver.ServerManager] [      Thread-13] activeMqExtendedMemory 512m ====== 512m
INFO  [com.optinuity.c2o.c2oserver.ServerManager] [      Thread-13] [Debug only]Starting ActiveMQ passing PID:6700, JAVA.HOME=e:\apps\java\jdk1.8.0_211\jre
INFO  [com.optinuity.c2o.c2oserver.ServerManager] [      Thread-13] ActiveMQ completed with exitStatus: 0
ERROR [com.optinuity.c2o.c2oserver.ServerManager] [      Thread-13] Unable to start ActiveMQ or it has been stopped...
ERROR [com.optinuity.c2o.c2oserver.ServerManager] [      Thread-13] PAM Server will shut down now

Environment

Process Automation 4.3.x

Cause

With this information, ActiveMQ logs (activemq.log) must be reviewed, and the following message may be seen:

ERROR | Failed to load: file [E:\Apps\PAM\activemq\conf\broker-config.xml], reason: Error creating bean with name 'org.apache.activemq.xbean.XBeanBrokerService#0' defined in file [E:\Apps\PAM\activemq\conf\broker-config.xml]: Invocation of init method failed; nested exception is java.lang.RuntimeException: Failed to start job scheduler store: JobSchedulerStore: activemq-data\Node0\scheduler | org.apache.activemq.xbean.XBeanBrokerFactory | main

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.apache.activemq.xbean.XBeanBrokerService#0' defined in file [E:\Apps\PAM\activemq\conf\broker-config.xml]: Invocation of init method failed; nested exception is java.lang.RuntimeException: Failed to start job scheduler store: JobSchedulerStore: activemq-data\Node0\scheduler

Caused by: java.lang.RuntimeException: Failed to start job scheduler store: JobSchedulerStore: activemq-data\Node0\scheduler

Caused by: java.io.FileNotFoundException: activemq-data\Node0\scheduler\scheduleDB.redo (The file or directory is corrupted and unreadable)

Resolution

The error seen on logs means that after the abnormal stop, ActiveMQ database got corrupted and is unable to be used.

The easy way to solve this is to stop Orchestrator service, verify that no Java.exe is running on the server, then navigate to ..\CA\PAM\activemq and rename "activemq-data" to something else.

Now start Orchestrator service again and ActiveMQ database should get recreated.

Additional Information

If this doesn't help, probably would be better to "Reinstall" Process Automation as a way to let the installer re create all ActiveMQ folders.