System health failed - Data Collector

Products

• CA Performance Management - Usage and Administration
• DX NetOps

Issue/Introduction

Until now all works fine, except the notifications about the system health.

Still appearing the NFA and Data Collector errors, this "down" in the components and Data source, happened in the upgrade procedure, but I don't found the way to reset or clear the notifications.

 

 

Environment

Release : 21.2.x

Component : IM Reporting / Admin / Configuration

Cause

The CollectorState is Unknown in the following RESTful query:

http://DA:8581/rest/dcms

 

-<DataCollectionMgrInfo version="1.0.0">

<ID>8743</ID>

<DcmID>datacollector:849dec33-30f6-49a0-8bce-eecc0a2d64ba</DcmID>

<Enabled>true</Enabled>

<IPAddress>192.168.0.15</IPAddress>

<RelatedDeviceItem>8744</RelatedDeviceItem>

<HostName>datacollector</HostName>

<MonitoredDeviceCount>16</MonitoredDeviceCount>

<CollectorState>UNKNOWN</CollectorState>

<IPAddressType>4</IPAddressType>

<TenantName>Default Pseudo Tenant</TenantName>

 

-<DataCollectionMgrInfo version="1.0.0">

<ID>684661</ID>

<DcmID>yyyyyyyyyyyyyy:25630dc2-1185-4735-aec3-294a49d2ace5</DcmID>

<Enabled>true</Enabled>

<IPAddress>XXX.XXX.XXX.XXX</IPAddress>

<RelatedDeviceItem>684662</RelatedDeviceItem>

<HostName>yyyyyyyyyyyyy</HostName>

<MonitoredDeviceCount>586</MonitoredDeviceCount>

<CollectorState>UNKNOWN</CollectorState>

<IPAddressType>4</IPAddressType>

<TenantName>Santander</TenantName>

 

The port 61616 should not be associated to the loopback address (127.0.0.1).

 [[email protected] Spectrum]# netstat -an | grep 616
tcp6       0      0 :::61617                :::*                    LISTEN
tcp6       0      0 :::61618                :::*                    LISTEN
tcp6       0      0 :::61619                :::*                    LISTEN
tcp6       0      0 :::61620                :::*                    LISTEN
tcp6       0      0 :::61621                :::*                    LISTEN
tcp6       0      0 :::61622                :::*                    LISTEN
tcp6       0      0 :::61623                :::*                    LISTEN
tcp6       0      0 127.0.0.1:61616         :::*                    LISTEN

 

Network Flow Analysis notification is because there is an issue with NFA itself. Raise a NFA ticket.

Resolution

We checked the activemq.log on the data collector and saw that it was trying to connect to the old DA name.

On the DC, We located and edited:
/opt/IMDataCollector/broker/apache-activemq-5.16.2/conf/activemq.xml

And updated all 4 references of the old DA.

After recycling the DCMD and activemq services, the DC’s connected to the DA and were green in the console.