Polled Item Count is zero after Performance Management upgrade
search cancel

Polled Item Count is zero after Performance Management upgrade

book

Article ID: 204956

calendar_today

Updated On:

Products

CA Infrastructure Management CA Performance Management - Usage and Administration DX NetOps

Issue/Introduction

After upgrading Performance Management to r20.2.x the Data Collector shows 0 (zero) polled items and does not collect data.

The following error is seen when the problem is being seen. It's found in the Data Aggregator karaf.log found in (default path) /opt/IMDataAggregator/apache-karaf-<version>/data/log.

The item count referenced in the error message aligns with the number of devices discovered and managed by the Data Aggregator.

ERROR | ository-thread-1 | 2020-12-10 01:45:41,382 | ExceptionLog | .ca.im.core.util.ExceptionLogger   99 | m.ca.im.common.core.util |       | A NEW application exception occurred (Key=e3afd22585b555740577cbf3d6e80796e0737f46) : Failed to load 31864 items of type {http://im.ca.com/inventory}Device on listener com.ca.im.dm.pollcfgctlr.impl.PccPollingConfigListener
java.lang.NullPointerException
        at com.ca.im.dm.pollcfgctlr.impl.ConsolidatedMonitoringProfileObject.populatePolledMetricsMap(ConsolidatedMonitoringProfileObject.java:477)[238:com.ca.im.aggregator.discovery:20.2.5.RELEASE-450]
        at com.ca.im.dm.pollcfgctlr.impl.ConsolidatedMonitoringProfileObject.getPolledMetrics(ConsolidatedMonitoringProfileObject.java:413)[238:com.ca.im.aggregator.discovery:20.2.5.RELEASE-450]
        at com.ca.im.dm.pollcfgctlr.impl.MonitoredMetricsListener.processCmpAndMF(MonitoredMetricsListener.java:149)[238:com.ca.im.aggregator.discovery:20.2.5.RELEASE-450]
        at com.ca.im.dm.pollcfgctlr.impl.MonitoredMetricsListener.processChangesOnMF(MonitoredMetricsListener.java:134)[238:com.ca.im.aggregator.discovery:20.2.5.RELEASE-450]
        at com.ca.im.dm.pollcfgctlr.impl.MonitoredMetricsListener.attributeChanged(MonitoredMetricsListener.java:87)[238:com.ca.im.aggregator.discovery:20.2.5.RELEASE-450]
        at com.ca.im.dm.pollcfgctlr.impl.PccPollingConfigListener.migratePCCMonitoredMetrics(PccPollingConfigListener.java:390)[238:com.ca.im.aggregator.discovery:20.2.5.RELEASE-450]
        at com.ca.im.dm.pollcfgctlr.impl.PccPollingConfigListener.performPollableAttrMigration(PccPollingConfigListener.java:329)[238:com.ca.im.aggregator.discovery:20.2.5.RELEASE-450]
        at com.ca.im.dm.pollcfgctlr.impl.PccPollingConfigListener.onLoad(PccPollingConfigListener.java:295)[238:com.ca.im.aggregator.discovery:20.2.5.RELEASE-450]
        at com.ca.im.item.impl.repository.NotificationManager.notifyLifecycleListenerOnLoad(NotificationManager.java:472)[204:com.ca.im.item-repository.impl:20.2.5.RELEASE-450]
        at com.ca.im.item.impl.repository.NotificationManager$3.run(NotificationManager.java:630)[204:com.ca.im.item-repository.impl:20.2.5.RELEASE-450]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[:1.8.0_222]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[:1.8.0_222]
        at java.lang.Thread.run(Thread.java:748)[:1.8.0_222]

Environment

DX NetOps Performance Management release r20.2.x releases

Cause

The Monitoring Profile facettypes are empty as far as the database iRep is concerned. This is despite them existing in the system.

The cause for that taking place, the true root cause, is yet to be determined.

Engineering continues to investigate via defect DE487180.

Resolution

If the problem is encountered during an upgrade to a DX NetOps Performance Management release r20.2.5 or earlier, open a new case with Support. Reference this Knowledge Base article. A pair of custom scripts will be provided to resolve the problem. The high level steps at that point will be:

  1. Stop the Data Aggregator
  2. Complete the upgrade for all systems if not yet completed.
  3. Run the provided clear script against the Data Aggregator while it's stopped.
  4. Start the Data Aggregator
  5. Once the Data Aggregator completes it's start up, run the set script.

This will be resolved in the r20.2.6 where the code is improved to ensure the problem doesn't take place.

Current recommendation is to avoid an upgrade to r20.2.5, wait for the r20.2.6 release which will prevent the problem.

Additional Information

Run the following query to determine if the issue will be seen during upgrade. The query MUST return 0 rows. If it returns anything other than 0 (zero) rows, do not proceed with the upgrade. Open a new support case for resolution before running the upgrade. Proceeding with an upgrade with a result >0 (zero) rows WILL result in significant down time.

select distinct long_value from v_list_attribute_instance where attr_qname='{http://im.ca.com/monitoringprofile}ConsolidatedMonitoringProfile.SourceMonitoringProfileID' and long_value NOT in (select distinct item_id from v_item_facet where facet_qname='{http://im.ca.com/monitoringprofile}MonitoringProfile') order by 1;

Powered by Wolken Software