Duplicate Data Collectors after upgrading

Products

• Network Observability
• CA Performance Management

Issue/Introduction

After upgrading NetOps we are seeing duplicated Data Collectors in NetOps Portal for the same IPAddress however they have different ID and Version.

Environment

Network Observability DX NetOps Performance Management Data Collectors

Cause

During an upgrade from 25.4.3 and earlier releases using apache-karaf, to releases 25.4.4 and newer using springboot, an incorrect DCM_ID leads to an upgrade problem causing duplicate DC entries.

The collector-manager-id variable value in the $DC_HOME/apache-karaf/etc/com.ca.im.dm.core.collector.cfg file is incorrect.

It should be using a value that shows HOSTNAME:UUID

• Some instances show a DCM_ID that only contains the UUID. No hostname precedes it.
• Some instances show a DCM_ID that is just a seemingly random number. No hostname or UUID.

The correct format should be "<hostName>:<long_ID_String>".

• The <hostName> is normally the first host the DC was ever installed on.
• The ID string is a unique value per DC install.

Resolution

Use the following solution if:

• The DCM_ID syntax issue is discovered before the upgrade. Fix it before upgrading using these steps.
  • Resolving this before hand avoids the duplicate DC entries.
• The DCM_ID syntax issue is discovered post upgrade. Fix it after the upgrade using these steps.
  • Resolving this after the upgrade? When the DC is restarted at the end of the process, when it reconnects to the DA it should result in removal of the 'duplicate' DC entry.

Follow these steps to resolve the issue.

1. Determine the correct DCM_ID and DC ID to use in the process.
   • Determine the correct DCM_ID.
     • Look in the $DC_HOME/broker/apache-activemq*/conf/activemq.xml.
     • Find the brokerName entry in the file. What does it show?
     • Will look similar to:

       • brokerName="dc_broker_UUID"

          

   • Determine the correct DC Item ID to use in the vsql queries below.
     • Go to DA_HostName:<Port>/rest/dcms
       • Note the <ID> value for each DC.
2. Stop the DA dadaemon service.
   1. The activemq service can remain running.
   2. Run this to stop the DA:
      1. Single DA? Run
         • systemctl stop dadaemon
      2. FT DA? Run:
         • $DA_HOME/scripts/dadaemon maintenance
         • Do this for both DA's to ensure both are shut down.
3. Connect to the vsql prompt on the Vertica DB.
   1. Run this vsql command for each DC needing a new DCM_ID value.
      • update <schema>.attribute_instance set string_value='<DCM_ID>' where attribute_id=(select attribute_id from <schema>.v_attribute where attr_qname='{http://im.ca.com/dcm}DataCollectionMgrInfo.DcmID' limit 1) and item_id=<ID>;
        • Set <DCM_ID> to the correct value from activemq.xml on the DC.
        • Set <ID> to the <ID> from our DA /rest/dcms data.
        • Run \d in the vsql prompt. Set the schema name in <schema>.
   2. Run this when all update commands are run, before exiting vsql.
      • commit;
4. After the vsql commands are completed start the DA.
   1. Run this to start the DA:
      1. Single DA? Run:
         • systemctl start dadaemon".
      2. FT DA? Run
         • $DA_HOME/scripts/dadaemon activate
         • Do this for both DA's to ensure both are ready for use.
   2. Ensure the DA returns to Green/Available in Portal.
      • Monitor Administration -> Data Sources -> Data Sources
5. Stop the DC
   1. The activemq service can remain running. 
   2. Run this to stop the DC.
      • systemctl stop dcmd
6. Edit the $DC_HOME/apache-karaf/etc/com.ca.im.dm.core.collector.cfg.
   1. Set collector-manager-id=HOSTMNAME:UUID
   2. Set it using the value set in the vsql queriy <DCM_ID>
   3. Save the changes.
7. Start the DC
   1. Run this to start the DC:
      1. systemctl start dcmd
8. After a few minutes validate the DC shows as Green and fully connected in Portal UI.
   • Monitor Administration -> Monitored Items Management -> Data Collectors
9. Confirm the DC shows the correct new DCM_ID in the ID column and that it's polled devices list looks correct.