Change Detection didn't find interface changes
search cancel

Change Detection didn't find interface changes

book

Article ID: 203124

calendar_today

Updated On:

Products

CA Performance Management - Usage and Administration DX NetOps

Issue/Introduction

It was brought to our attention that a specific interface was not being monitored in PM, we checked Spectrum and the interface was present and according to the configuration had been added in late August this year. However PM didn't have the interface listed, we tried rediscovering the device, that didn't work, so we rediscovered the interface MF and that finally added the interface into PM. As the interface is part of a monitoring profile that is "12 hour change detection" unsure why this wasn't picked up.

We would like to investigate why this didn't work as we expected, and if possibly we are missing others due to something not working as expected.

DX NetOps Performance Management Data Aggregator fails to discover new interfaces.

DX NetOps Performance Management Data Aggregator fails to find and update existing interfaces with value changes.

DX NetOps Performance Management Portal web UI shows old out of data ifAlias values for interfaces. Spectrum and the device MIB show the correct current values for ifAlias.

Environment

All supported DX NetOps Performance Management releases

Cause

Change Detection (CD) has failed, going into a disabled state, for the Interface Metric Family that manages the devices interface items.

This is observed when the following messages are seen in the DA karaf.log file (default home path is /opt/IMDataAggregator/apache-karaf<version>/data/log). They can be seen once every time CD runs. Default time frame is every 24 hours.

INFO  | er-thread-135572 | 2024-12-11T01:36:23,143 | nagedDeviceResourceDiscoveryImpl | nagedDeviceResourceDiscoveryImpl  687 | .im.aggregator.discovery |       | Change detection for MF NormalizedPortInfo on device 51702397 is disabled.
INFO  | er-thread-135572 | 2024-12-11T01:36:23,143 | nagedDeviceResourceDiscoveryImpl | nagedDeviceResourceDiscoveryImpl  626 | .im.aggregator.discovery |       | The discovery of {http://im.ca.com/normalizer}NormalizedPortInfo at device 51702397 was  disabled.  Bailed out.

Use the Item_ID set on the device that hosts the problem interface to find the messages in the karaf.log.

  • The device is represented by it's Item_ID value found in the DR DB. To easily find it for use in searching the karaf.log:
    • In Portal go to Administration -> Monitored Items Management -> Monitored Devices.
    • Find the problem device. Select it and review it's Details tab.
    • Note the value in the "Item ID" field.
    • Use that to search the karaf.log file.
  • In this example NormalizedPortInfo is the internal VC name for the Interface Metric Family. To find the internal name expose them in the UI. 
    • In Portal go to Administration -> Data Sources -> Data Aggregator.
    • Under Monitoring Configuration select Metric Families.
    • Mouse over the Name column in the Metric Families list.
    • Select the Gear icon and select the Internal Name column to be displayed.
    • Optional: Repeat and choose Save Column Settings to save the column changes

Resolution

To resolve this in the Portal UI take the following steps.

  1. Find and select the device hosting the interface in the DA Monitored Devices Inventory.
    1. In Portal go to Administration -> Monitored Items Management -> Monitored Devices.
    2. Find the problem device. Select it and review it's Polled Metric Families tab.
  2. Resolve the issue:
    1. Select the line representing the Interface MF.
    2. Select the Update Metric Family button.
  3. That will re-enable CD for the Interface MF on this device. When it completes the Status value will move from Discovering back to Supported.
  4. When it completes confirm the changes missing were made.

 

To manage this using REST API calls review the following.

  • Run "Update Metric Family" on target devices and metric families using REST.
  • Check a REST endpoint on the DA to monitor and check and correct any devices that are disabled:
  • In all URLs replace:
    • <scheme> with your scheme in use: HTTP or HTTPS
    • <DA_HostName> with the name of your DA host
    • <Port> with the DA port in use. Default is 8581 for HTTP. Recommended HTTPS port is 8582.

Which devices have disabled Metric Family entries?

Use a REST call set to POST to the URL:

  • <scheme>://<DA_HostName>:<Port>/rest/devices/filtered

In the call BODY add the following content.

<FilterSelect xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
              xsi:noNamespaceSchemaLocation="filter.xsd">
	<Filter>
		<Device.ChangeDetectionDisabledMFs type="CONTAINS">{http://im.ca.com/normalizer}</Device.ChangeDetectionDisabledMFs>
	</Filter>
	<Select use="exclude"
	        isa="exclude">
		<Device use="exclude">
			<PrimaryIPAddress use="include"/>
			<ChangeDetectionDisabledMFs use="include"/>
		</Device>
		<Item use="exclude">
			<Name use="include"/>
		</Item>
	</Select>
</FilterSelect>


While the result set is empty, this shows an example result using the ARC REST API client.

A working result set with data will return entries for each device that look similar to this example. This gives us the devices Name, ID and the Metric Family or Families at issue.

<DeviceList>
  <Device version="1.0.0">
    <ID>14005</ID>
    <PrimaryIPAddress>10.1.1.1</PrimaryIPAddress>
    <ChangeDetectionDisabledMFsList>
      <ChangeDetectionDisabledMFs>{http://im.ca.com/normalizer}NormalizedPortInfo</ChangeDetectionDisabledMFs>
    </ChangeDetectionDisabledMFsList>
    <Item version="1.0.0">
      <Name>device A name</Name>
    </Item>
  </Device>
</DeviceList>


To resolve those you can run Change Detection on the Device and it's MF in the UI (see above) or use these CURL calls to the DA that will also trigger the updates.

The CURL syntax is the following. Replace <Device_Item_ID> with the DA Item ID value for the target device. Replace <adminRoleUserName> with a real username. You'll be prompted for the password when the command is run.

  • curl -H 'Content-Type: application/xml' -u <adminRoleUserName> -d '<emptybody/>' -X POST <scheme>://<DA_HostName>:<Port>/genericWS/devices/<Device_Item_ID>/metricfamilies/<MF_NAME>/discovery

An example for the Interface MF on device whose Item_ID is 14005 using the default admin user would be:

  • curl -H 'Content-Type: application/xml' -u admin -d '<emptybody/>' -X POST <scheme>://<DA_HostName>:<Port>/genericWS/devices/14005/metricfamilies/NormalizedPortInfo/discovery

Additional Information

To get into a state where we disable change detection, the following needs to happen.

  1. When update MF runs (either manually or via change detection schedule), we first discover what VC to use.
  2. We will read the OIDs needed for component discovery.
    1. If we determine that the VC(s) supported changed on the metric family, we kick off a FIRST_VC_Verify stage that runs in 5 mins from the initial read.
  3. After we get back the OIDs needed for component discovery, we compare them to the existing components.
    1. If we determine this will result in Not Present items being created, we do a SECOND_VC_VERIFY stage that runs in 5 mins from the first verify.
    2. If we determine this will still result in Not Present items being created, we disable change detection.
      1. We also do not apply any change/new item info we read during this CD cycle.
      2. When this occurs we print to the DA karaf.log.
        1. NormalizedPortInfo is an example MF name, and ITEMID will be the device item_id in DA.
        2. Change detection for NormalizedPortInfo with VC {...}IfXTableMib on device ITEMID is disabled.
Powered by Wolken Software