Data Aggregator fails synchronization

book

Article ID: 224835

calendar_today

Updated On:

Products

CA Performance Management - Usage and Administration DX NetOps

Issue/Introduction

Received messages in CAPC Portal Dashboard: 'Source is currently unavailable (query ID....) '

Found failures on CAPC for DA Data Source synchronization in CAPC data source logs.

Manually running test on the datasource passed, but re-sync on the DA did not succeed.

Stopped and restarted the DA daemon and ActiveMQ and then was able to do a full resync.

Why do we see Data Aggregator synchronization failures intermittently in DX NetOps Performance Management PC Portal?

When those synchronization failures occur we also see these errors in the Device Manager DMService.log files.

ERROR | pool-4-thread-6          | 2021-09-12 16:35:41,307 | com.ca.im.portal.dm.productsync.DataSourcePoller|
|
Data source Data [email protected] encountered an error while processing a sync request. The problem may be in either the data source or CAPC.
The following stack trace may indicate where the problem lies.
You may also want to check the logs for the data source to determine if it is the cause of the problem.
java.lang.NullPointerException

Cause

Unable to be determined. Synchronization failures stopped appearing.

Environment

All supported DX NetOps Performance Management releases

Resolution

To capture more meaningful messages next time synchronization failures related to the NullPointerException error occur make the following changes.

This will not change the logging for DataSourcePoller messages in the normal DMService.log and wrapper-<date>.log files. Instead it will add a new DataSourcePoller.log file. The hope is that these changes will help capture the full stack trace we're missing in the current logs to identify the cause of the NullPointerException.

To make the change open the (default path) /opt/CA/PerformanceCenter/DM/resources/log4j.properties file for editing. Add this section to the end of the file and save the changes.

#
# Capture full stack trace for DA sync failures generating NullPointerException errors in DMService.log.
# Generates new DataSourcePoller.log file in the (default path) /opt/CA/PerformanceCenter/DM/log directory.
# 
log4j.appender.ExceptionLog=org.apache.log4j.RollingFileAppender
log4j.appender.ExceptionLog.layout=org.apache.log4j.PatternLayout
log4j.appender.ExceptionLog.layout.ConversionPattern=%d | %p | %c | (%F:%L) %n    | %m %n
log4j.appender.ExceptionLog.file=/opt/CA/PerformanceCenter/DM/logs/DataSourcePoller.log
log4j.appender.ExceptionLog.MaxFileSize=100MB
log4j.appender.ExceptionLog.MaxBackupIndex=1
log4j.logger.com.ca.im.portal.dm.productsync.DataSourcePoller=DEBUG, ExceptionLog, stdout, application-timesize

After making the change to the log4j.properties file we need to restart the DM and PC console services.

For RH 7.x use these commands:

  • systemctl stop caperfcenter_console
  • systemctl stop caperfcenter_devicemanager
  • systemctl start caperfcenter_devicemanager
  • Wait 30 seconds...
  • systemctl start caperfcenter_console

For RH 6.x use these commands:

  • service caperfcenter_console stop
  • service caperfcenter_devicemanager stop
  • service caperfcenter_devicemanager start
  • Wait 30 seconds...
  • service caperfcenter_console start

When we see new instances of the errors, gather a diagnostics logging package from the PC Portal host using the re.sh script. Attach the resulting logging package to a new support case requesting assistance finding a cause and solution for the failures.

Instructions to run the re.sh script are found in the Unable to Resolve Issue documentation.