After upgrading to CA Performance Management 3.7.10, the Data Aggregator consumes 100% CPU and remains there
Article ID: 188059
Updated On:
Products
Issue/Introduction
After upgrading to CA Performance Management 3.7.10, the Data Aggregator consumes 100% CPU and remains there.
Virtual Network Assurance is integrated with VNA
Environment
Release : 3.7
Component : IM Reporting / Admin / Configuration
Resolution
This issue is fixed in CA Performance Management 3.7.11:
Symptom: After upgrading to 3.7.10 multiple symptoms were seen on certain Data Aggregators that were monitoring SDWAN environments with VNA. 1. High CPU Usage after upgrade caused PC to lose contact with the DA ( see: https://knowledge.broadcom.com/external/article?articleId=188059 ). 2. Changes in the monitored VNA environment were not being reflected in CAPC. These include, but may not be limited to: Sites, Tunnels and SLAPaths. 3. Prior to and after upgrading to 3.7.10 "duplicate" / "stale" items and SDN Domains were seen in PC under the SDN Domains group.
Resolution: The Data Aggregator's processing of VNA data has been updated to be more efficient during the processing of data from SDWAN environments. The changes include:1. A reduction in the upper limit of the number threads available for SDN processing to 1/3 of the CPUs on the Data Aggregator. 2. General improvements to the VNA processing to improve the efficiency of processing VNA updates from SDWAN environments. 3. Additional protections have been added to the VNA processing to resolve missing items seen after upgrading. 4. SDN processing in the DA has been updated to prevent Items from being created for SDN Gateways that had already been deleted from the DA.
(3.7.11, DE455285)
At this time there is a workaround as follows:
1) Back up the following file:
/opt/IMDataAggregator/apache-karaf-2.4.3/bin/setenv
2) In file:
/opt/IMDataAggregator/apache-karaf-2.4.3/bin/setenv
Change
if [ "$KARAF_OPTS" == "" ]; then
export KARAF_OPTS="-XX:+HeapDumpOnOutOfMemoryError -Dorg.apache.activemq.SERIALIZABLE_PACKAGES=*"
fi
To:
if [ "$KARAF_OPTS" == "" ]; then
export KARAF_OPTS="-XX:+HeapDumpOnOutOfMemoryError -Dorg.apache.activemq.SERIALIZABLE_PACKAGES=* -Djava.util.concurrent.ForkJoinPool.common.parallelism=4"
fi
2) Restart the Data Aggregator
systemctl stop dadaemon
systemctl start dadaemon
Note: This change would be overwritten by any updates/upgrades of the Data Aggregator as this file is rebuilt in that process. So no need to remove when updating to CA Performance Management 3.7.11 or later.
Additional Information
Feedback
