ActiveMQ on Data Aggregator of DX NetOps Performance Management failing to start

Products

• CA Performance Management
• Network Observability

Issue/Introduction

After upgrading DX NetOps Performance Management, the Data Aggregator (or Data Collector) upgrade installation shows: Successful with errors.

This can also be seen after server patching. Something in the system updates for the server resulted in the file being modified.

The primary symptom is the ActiveMQ service fails to stay running. After start it just shows a generic failure state in systemctl status output. The same generic error is seen in /var/log/messages.

Unable to start ActiveMQ service on Data aggregator

The systemctl status output might show something like this in it's output.

Jun 22 19:39:19 host systemd[1]: activemq.service: main process exited, code=exited, status=1/FAILURE
Jun 22 19:39:19 host systemd[1]: Unit activemq.service entered failed state.
Jun 22 19:39:19 host systemd[1]: activemq.service failed.

The /var/log/messages file would contain messages similar to this, matching systemctl status output as well as additional details.

Nov 23 03:55:26 <DA_Host> activemq[1258556]: Starting Apache ActiveMQ.
Nov 23 03:55:26 <DA_Host> activemq[1258586]: INFO: Loading '/opt/IMDataAggregator/broker/apache-activemq-6.1.6/bin/setenv'
Nov 23 03:55:26 <DA_Host> activemq[1258586]: INFO: Using java '/opt/IMDataAggregator/jre/bin/java'
Nov 23 03:55:26 <DA_Host> activemq[1258586]: INFO: Starting - inspect logfiles specified in logging.properties and log4j2.properties to get details
Nov 23 03:55:26 <DA_Host> activemq[1258683]: INFO: pidfile created : '/opt/IMDataAggregator/broker/apache-activemq-6.1.6/data/activemq.pid' (pid '1258684')
Nov 23 03:55:26 <DA_Host> systemd[1]: Started Apache ActiveMQ.
Nov 23 03:55:26 <DA_Host> systemd[1]: activemq.service: Main process exited, code=exited, status=1/FAILURE
Nov 23 03:55:26 <DA_Host> systemd[1]: activemq.service: Failed with result 'exit-code'.
Nov 23 03:56:23 <DA_Host> systemd[1]: Configuration file /etc/systemd/system/activemq.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.

In the //activemq.out file we see the following messages:

Picked up JAVA_TOOL_OPTIONS: -Dcom.sun.management.jmxremote.ssl.need.client.auth=false
Error: Password file read access must be restricted: /opt/IMDataAggregator/broker/apache-activemq-6.1.6/conf/jmx.password
jdk.internal.agent.AgentConfigurationError
        at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.checkPasswordFile(Unknown Source)
        at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(Unknown Source)
        at jdk.management.agent/jdk.internal.agent.Agent.startAgent(Unknown Source)
        at jdk.management.agent/jdk.internal.agent.Agent.startAgent(Unknown Source)

Checking the referenced file we see:

 -rw----r-x 1 root root 473 Nov 22 01:09 /OPT/IMDataAggregator/broker/apache-activemq-6.1.6/conf/jmx.password

Environment

All supported Network Observability DX NetOps Performance Management Data Aggregator releases

Cause

Permissions on the /opt/broker/apache-activemq/conf/jmx.password file are incorrect.

Something during monthly patching, or during the upgrade run, incorrectly modified the files permissions.

Resolution

To resolve this change permissions on the jmx.password file. It must ONLY allow the root user any form of access.

To resolve this, run the following command as the root user. The default install home is shown, change the path as needed.

chmod 600 /opt/broker/apache-activemq/conf/jmx.password

After setting that when checking file permissions we should now see the following.

-rw------- 1 root root 473 Nov 22 01:09 /opt/IMDataAggregator/broker/apache-activemq-6.1.6/conf/jmx.password

Once modified restart activemq and the dadaemon if it's not running.