Convert Fault Tolerant Data Aggregator to Standalone Data Aggregator due to environment issues

Products

• CA Performance Management - Usage and Administration
• DX NetOps

Issue/Introduction

After configuring a Fault Tolerant Data Aggregator environment, due one of many possible reasons the given environment is not capable of supporting a stable Fault Tolerant configuration.

In that case it may be necessary to change back to a standalone Data Aggregator configuration that the environment is more capable of supporting in a stable manner.

Environment

All supported DX NetOps Performance Management releases

Cause

Environmental instability such as a network issue that prevents steady data flow. Instability can also occur due to issues with shared disks. So the problems only occur due to the environment, not the application itself.

Resolution

1. Shut down all PC services. Stop them in the following order, using the following commands:
   systemctl stop caperfcenter_console
   systemctl stop caperfcenter_devicemanager
   systemctl stop caperfcenter_eventmanager
   systemctl stop caperfcenter_sso
   
   
2. Shut down the proxy host daproxy and consul services using the following commands. Start/stop order doesn't matter.
   

   systemctl stop consul

   systemctl stop daproxy
   
   

3. Run the following command to set the Data Aggregator that will remain in place to Maintenance so it's not running during the changes. Run the command from the directory (default path)
   

   cd /opt/IMDataAggregator/scripts/
   ./dadaemon maintenance
   
   

4. Copy all files from the DASharedRepo directory to /opt/IMDataAggregator/data.
   

   cp -rfp <DASharedRepo> /opt/IMDataAggregator/data 

   
   Make sure the files are owned by the install/run user of the DA. If not root, then run:
   

   chown -R <user> /opt/IMDataAggregator/data 
   
   

5. Make a backup copy of /etc/DA.cfg and set the following changes. First make a backup, something like:
   

   cp DA.cfg DA.cfg.origFTConfig

   
   Now edit the file and set the following variables with the values seen:
   

   da.ha.configured=no
   da.proxy.host=
   da.consul.port=
   da.data.home=/opt/IMDataAggregator/data
   da.data.home.fstype=
   
   

6. Then update the file:
   

   /opt/IMDataAggregator/broker/apache-activemq-*/conf/activemq.xml

   
   to change the directory to point to: /opt/IMDataAggregator/data/broker/kahadb
   
   

   <kahaDB checkForCorruptJournalFiles="true"
   checksumJournalFiles="true"
   directory="/opt/IMDataAggregator/data/broker/kahadb"
   ignoreMissingJournalfiles="true"/>
   
   

7. Uninstall the Consul and Consul-ext services on the DA that will remain in use. Go to (default path):
   

   cd /opt/IMDataAggregator/Uninstall_Consul

   
   As install owner, Run:
   

   ./Uninstall

   
   Go to (default path):
   

   cd /opt/IMDataAggregator/Uninstall_Consul-ext

   
   As install owner, Run:
   

   ./Uninstall
   
   

8. On the Data Collector(s) make a backup copy of /opt/DCM.cfg file and set the following changes. First make a backup, something like:
   

   cp DCM.cfg DCM.cfg.origFTConfig

   
   Now edit the file and set the following variables with the values seen:
   

   IS_DA_HA_CONFIGURED=no
   DA_HOST=<1 DA>
   DA_STANDBY_HOST=<1 DA>
   
   

9. On the Data Collector(s) make a backup copy of the /opt/IMDataCollector/broker/apache-activemq-5.15.8/conf/activemq.xml file. Use something like:
   

   cp activemq.xml activemq.xml.origFTConfig

   
   Edit the file and find the four lines showing the following:
   
   

   <networkConnector name="da_manager" uri="static:(failover:(tcp://DA_A:61616,tcp://DA_B:61616)?maxReconnectAttempts=3)" duplex="true" suppressDuplicateTopicSubscriptions="false">
   <networkConnector name="da_manager-PRQ" uri="static:(failover:(tcp://DA_A:61618,tcp://DA_B:61618)?maxReconnectAttempts=3)" duplex="true" suppressDuplicateTopicSubscriptions="false">
   <networkConnector name="da_manager-IREP" uri="static:(failover:(tcp://DA_A:61620,tcp://DA_B:61620)?maxReconnectAttempts=3)" duplex="true" suppressDuplicateTopicSubscriptions="false">
   <networkConnector name="da_manager-blob" uri="static:(failover:(tcp://DA_A:61622,tcp://DA_B:61622)?maxReconnectAttempts=3)" duplex="true" suppressDuplicateTopicSubscriptions="false">

   
   For each line, changed them to the following, replacing Remaining-DA_Host with the host name of the Data Aggregator that will remain.
   
   

   <networkConnector name="da_manager" uri="static:(tcp://Remaining-DA_Host:61616)" duplex="true" suppressDuplicateTopicSubscriptions="false">
   <networkConnector name="da_manager-PRQ" uri="static:(tcp://Remaining-DA_Host:61618)" duplex="true" suppressDuplicateTopicSubscriptions="false">
   <networkConnector name="da_manager-IREP" uri="static:(tcp://Remaining-DA_Host:61620)" duplex="true" suppressDuplicateTopicSubscriptions="false">
   <networkConnector name="da_manager-blob" uri="static:(tcp://Remaining-DA_Host:61622)" duplex="true" suppressDuplicateTopicSubscriptions="false">

   
   Save the changes to the file.
   
   
10. On the Data Aggregator run the following command to start it properly as a standalone system.
    

    systemctl start dadaemon
    
    

11. On the Data Collector(s) stop dcmd and activemq services with the following commands, in the following order.
    

    systemctl stop dcmd
    systemctl stop activemq

    
    Start the dcmd service, which will start the activemq services, with the following command:
    

    systemctl start dcmd
    
    

12. Edit the Data Aggregator Data Source via the Manage Data Sources page in the web UI. Set the standalone Data Aggregator host name. Ensure a successfull communicaton Test runs against the remaining DA instead of the proxy. Once Test is successful, saved the changes and the DA should sync successfully after a bit of time has passed, normally within 30 minutes.
    
    
13. Uninstall the Consul and daproxy on the proxy host.