Data Aggregator Shows DR Nodes as Down in System Status - DX NetOps Performance Management

Products

• CA Infrastructure Management
• CA Performance Management
• Network Observability

Issue/Introduction

The DX NetOps Performance Management System Status page may display Data Repository (DR) nodes as "Down," while the underlying Vertica database confirms nodes are "UP" via vsql. This indicates the Data Aggregator (DA) is unable to receive heartbeat responses from the DR nodes within the default timeout threshold.

Symptom

The System Status page reports DR nodes as "Down" (e.g., node0001, node0002). The vsql command returns the status as "UP": SELECT node_name, node_state FROM nodes;

The karaf.log or shutdown.log contains heartbeat failure entries: DB heartbeat to host #### failed.

Data Aggregator keeps shutting down with the following error in the karaf.log file.

ERROR | t Monitor Thread | 2015-09-03 10:19:48,944 | shutdown | ase.heartbeat.DBStateManagerImpl 385 | ommon.core.services.impl | | DB heartbeat to host ><hostname of Db execeeded max non-success time of 300000
 
WARN | t Monitor Thread | 2015-09-03 10:19:48,944 | shutdown | ase.heartbeat.DBStateManagerImpl 726 | ommon.core.services.impl | | DB state for host <hostname of Db> changing from OK to DOWN

Environment

• DX NetOps Performance Management (All supported versions)
• Data Aggregator installed with default heartbeat settings

Cause

The Data Aggregator DBStateManager service marks nodes as "DOWN" if the heartbeat response takes longer than the default 5-minute (300,000ms) threshold. Performance latency in the database or network environment can trigger this state.

Resolution

The two solutions for each common cause are as follows.

1. To check if the Data Repository Vertica database is up and running, and restart if it needed, review this Knowledge Base Article regarding proper restart of the Performance Management environment.
   
   
2. If the database is found to be running, the slow time to complete the connection could be causing the issue. The Data Aggregator waits for 5 minutes before shutting itself down without a valid connection to the database. We can try to temporarily extend the time the Data Aggregator waits to complete this connection before giving up. Making the following change will raise the wait time to 10 minutes. To make this change:

1. 1. In the (default path) /opt/IMDataAggregator/apache-karaf-<version>/etc directory create a new file named:
      • com.ca.im.core.services.database.heartbeat.DBStateManager.cfg
        
        
   2. Within the new file add the following property entries to specify a 10 minute timeout failure threshold instead of the default 300000, which is 5 minutes.
      • maxNonSuccessTimeBeforeDRNodeConsideredDown=600000
        
        
   3. Restart the dadaemon service to apply changes:
      • • systemctl stop dadaemon
        • systemctl start dadaemon

Additional Information

If option 2 is required to resolve the issue it does indicate a problem that is worth investigation despite it functioning. If this is desired please open a new support case.

If the database is running, and option 2 doesn't resolve the problem, please open a new case with the support team for investigation.