search cancel

After upgrading to CA Performance Management 3.7.10, the Data Aggregator consumes 100% CPU and remains there

book

Article ID: 188059

calendar_today

Updated On:

Products

CA Infrastructure Management CA Performance Management - Usage and Administration DX NetOps

Issue/Introduction

After upgrading to CA Performance Management 3.7.10, the Data Aggregator consumes 100% CPU and remains there.

Virtual Network Assurance is integrated with VNA 

Environment

Release : 3.7

Component : IM Reporting / Admin / Configuration

Resolution

This issue is fixed in CA Performance Management 3.7.11:

Symptom: After upgrading to 3.7.10 multiple symptoms were seen on certain Data Aggregators that were monitoring SDWAN environments with VNA.  1. High CPU Usage after upgrade caused PC to lose contact with the DA ( see: https://knowledge.broadcom.com/external/article?articleId=188059 ).  2. Changes in the monitored VNA environment were not being reflected in CAPC.  These include, but may not be limited to: Sites, Tunnels and SLAPaths.  3. Prior to and after upgrading to 3.7.10 "duplicate" / "stale" items and SDN Domains were seen in PC under the SDN Domains group.

Resolution: The Data Aggregator's processing of VNA data has been updated to be more efficient during the processing of data from SDWAN environments.  The changes include:1. A reduction in the upper limit of the number threads available for SDN processing to 1/3 of the CPUs on the Data Aggregator.  2. General improvements to the VNA processing to improve the efficiency of processing VNA updates from SDWAN environments.  3. Additional protections have been added to the VNA processing to resolve missing items seen after upgrading.  4. SDN processing in the DA has been updated to prevent Items from being created for SDN Gateways that had already been deleted from the DA.
(3.7.11, DE455285)

At this time there is a workaround as follows:

1) Back up the following file:

/opt/IMDataAggregator/apache-karaf-2.4.3/bin/setenv


2) In file:

/opt/IMDataAggregator/apache-karaf-2.4.3/bin/setenv

Change

 

    if [ "$KARAF_OPTS" == "" ]; then
        export KARAF_OPTS="-XX:+HeapDumpOnOutOfMemoryError -Dorg.apache.activemq.SERIALIZABLE_PACKAGES=*"
    fi

To:

    if [ "$KARAF_OPTS" == "" ]; then
        export KARAF_OPTS="-XX:+HeapDumpOnOutOfMemoryError -Dorg.apache.activemq.SERIALIZABLE_PACKAGES=* -Djava.util.concurrent.ForkJoinPool.common.parallelism=4"
    fi

2) Restart the Data Aggregator 

systemctl stop dadaemon
systemctl start dadaemon

Note: This change would be overwritten by any updates/upgrades of the Data Aggregator as this file is rebuilt in that process. So no need to remove when updating to CA Performance Management 3.7.11 or later.

Additional Information

3.7.11 Fixed Issues:

https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/it-operations-management/performance-management/3-7/release-notes/fixed-issues.html#concept.dita_dffccc4e37aafcca90f52a0d259b75e6551f37ff_3710Fixes