Interface Bits metrics show incorrect data in Performance Management
search cancel

Interface Bits metrics show incorrect data in Performance Management

book

Article ID: 187808

calendar_today

Updated On:

Products

CA Infrastructure Management CA Performance Management - Usage and Administration DX NetOps

Issue/Introduction

We have a pair of internet routers where the graphs keep going up and down between 700 MB and 100 MB for the Bits metrics.
 
This is not correct according to the device itself when checked for throughput details.

We tried Stop Polling and Start Polling for the device but the problem remains.
 
A sample chart in this instance shows the following strange trend line with extreme peaks and valleys.
 
 

Environment

Reported against r3.6.x release

Cause

When examining the Detailed Poll Logging for the problem items using the DA:8581/dcdebug page, we see that the interfaces don't show even 5 minute poll cycles. Instead in this instance we see one run, then the next 8  minutes later. The next cycle runs 2 minutes later and the cycle repeats from there.
 
Due to the change in expected 5 minute delta, the data is calculated against the wrong delta time frame resulting in the see-saw patter seen.

Resolution

To resolve the problem a Data Collector restart was run. Stopping and restarting the Data Collector dcmd service resulted in properly rescheduled poll cycles and normal looking trend lines.
 
Unfortunately due to log rotation after delay in log collection for analysis, root cause was unable to be determined.

Additional Information

If seen again, the following details should be gathered for engineering analysis.

  1. Enable the following debug on the Data Collector (DC) managing the problem device(s).
    1. Open a terminal window to the Data Collector polling/managing the device debug was enabled for.
    2. Open the (default path) /opt/IMDataCollector/apache-karaf-2.4.3/etc/org.ops4j.pax.logging.cfg file for editing.
    3. Add the following two lines to the section that starts with a commented section header titled "Discovery Logging".
      • log4j.logger.com.ca.im.dm.snmp.collector.SnmpPoller=DEBUG,DiscoveryLog
      • log4j.additivity.com.ca.im.dm.snmp.collector.SnmpPoller=false
  2. Restart the Data Collector. In this case we just want to stop and start the dcmd service using:
    1. RH 6.x
      • service dcmd stop
      • service dcmd start
    2. RH 7.x
      • systemctl stop dcmd
      • systemctl start dcmd
  3. Once the DC is started go to the DCDebug page at DA_HOST:8581/dcdebug.
    1. When it's working post DC restart enter the IP Address and IP Domain for a sample device.
    2. Select the Polling Configuration option and the View Data button.
    3. Find the entry for the Interface Metric Family, normally named NormalizedPortInfo. It should look something like this, but with different ID values.
      • Polling Config Item ID=71309
      • Poll Group ID=806
      • Normalized Facet Type={http://im.ca.com/normalizer}NormalizedPortInfo
      • Certification Facet Type={http://im.ca.com/certifications/snmp}IfXTableMib
    4. Note the Poll Group ID value for the Interface Metric Family.
    5. Go to Enable/Disable for Detailed Poll Logging.
      1. Set the IP and Domain.
      2. Set the Filtering option to "Poll Group IDs"
      3. Enter the Poll Group ID found in step 4 above in the ID List field.
      4. Enable the logging
  4. Let the Detailed Poll Logging run for one hour and gather the logging.
    1. NOTE: Do not disable the logging before gathering it. It will be lost if that is done. 
    2. Go to the main DCDebug page.
    3. Enter the IP Address and IP Domain for the device debug is enabled against.
    4. Select the "Download all logs for IP" option
    5. Select the "Download As Zip" button
    6. Set aside the resulting Zip file to attach to the support case.
  5. Disable the DC logging enabled in step 1. Can remove the lines from the file, or add a comment ('#' symbol) to the start of the line to have it ignored.
  6. Create a fresh DC CARE package via the re.sh script. Instructions to run that are found here.
Attach the resulting DCDebug zip file, and the DC CARE package, to a new support case. Reference this Knowledge Base article when opening the case.
 

Attachments

Powered by Wolken Software