Why did the Data Aggregator shut down and/or will not start

Products

• CA Performance Management - Usage and Administration
• DX NetOps

Issue/Introduction

My DA went down and I would like to determine root cause

Environment

Dx NetOps Performance Management all versions

Cause

Data Repository Vertica database is down and not running. As a result the Data Aggregator dadaemon service shuts itself down to protect itself. It will not successfully restart until the database is restarted as well.

Resolution

You can review /opt/IMDataAggregator/apache-karaf-*/shutdown.log on the DA to confirm whether the DR is down, or not able to be contacted. Then do the following

1. Start vertica database, by starting admintools on the Data Repository and choose "Start Database"
2. Go to the Data Aggregator and start the dadaemon process: service dadaemon start or systemctl start dadaemon (rhel7)

Additional Information

The Data Aggregator will shut down when it can no longer contact any Data Repository hosts.

The Data Aggregator will not start if it can not contact any Data Repository hosts.

You can review /opt/IMDataAggregator/apache-karaf-*/shutdown.log on the DA to confirm whether the DR is down, or not able to be contacted

On the Data Aggregator, in the ${da.karaf.home}/shutdown.log you will see messages similar to the following:

WARN  | :DRHOSTNAME-01 | 2018-05-04 02:40:05,458 | shutdown | ase.heartbeat.DBStateManagerImpl  731 | ase.heartbeat.DBStateManagerImpl  731 | ommon.core.services.impl |       | DB state for host DRHOSTNAME-01 changing from OK to DOWN
ERROR | anager-thread-13 | 2018-05-04 02:40:05,468 | shutdown | ces.shutdown.ShutdownManagerImpl  131 | ces.shutdown.ShutdownManagerImpl  131 | ommon.core.services.impl |       | Shutting down the data aggregator.It was detected that no data repository nodes were contactable. The uncontactable hosts are:[DRHOSTNAME-01]
ERROR | anager-thread-14 | 2018-05-04 02:40:05,468 | shutdown | tTolerantDBConnectionManagerImpl  221 | tTolerantDBConnectionManagerImpl  221 | ommon.core.services.impl |       | No DB host name available.
INFO  | anager-thread-14 | 2018-05-04 02:40:05,469 | shutdown | tTolerantDBConnectionManagerImpl  376 | tTolerantDBConnectionManagerImpl  376 | ommon.core.services.impl |       | The primary host for database transactions is now set to null
ERROR | anager-thread-14 | 2018-05-04 02:40:05,469 | shutdown | tTolerantDBConnectionManagerImpl  179 | tTolerantDBConnectionManagerImpl  179 | ommon.core.services.impl |       | The primary data repository host 'DRHOSTNAME-01' is no longer available, and there are no available secondary hosts. Current Host Status: {DRHOSTNAME-01=DOWN}
 
You can get the location of da.karaf.home from the /etc/DA.cfg:

]$ grep 'da.karaf.home' /etc/DA.cfg
da.karaf.home=/opt/IMDataAggregator/apache-karaf-2.4.3