System Status Page shows Data Repository and Data Collector red when they are working

Products

• DX NetOps
• CA Performance Management - Usage and Administration

Issue/Introduction

The DX NetOps Performance Management (PM) Data Repository (DR) and Data Collector (DC) status values from the System Status Page in the NetOps Portal web UI show as Red/Down.

When we examine those systems they are operating properly, they are not down despite the warnings from the System Status page.

The polled data is still being properly gathered and inserted into the database for reports.

The NetOps Portal web UI System Status page derives the status values for the DR and DC systems from the Data Aggregator (DA) system via REST API calls.

For the DC we use this API URL:

http(s)://daHostname:8581(2)/rest/dcms

For the DR we use this API URL:

http(s)://daHostname:8581(2)/rest/datarepositorynodes

When we try these manually in a browser they return 404 Not Found errors instead of the expected status information.

Environment

All supported DX NetOps Performance Management releases

Cause

Unknown. Root cause has not been identified yet. Issue was not reproducible with debug enabled.

Resolution

The workaround to this problem is restarting the REST web services for the DA. This can only be done by restarting the DA dadaemon service.

For a standalone DA stop the dadaemon service and restart it. After the DA completely restarts and shows Green/Normal in Portal post restart, the issue should be resolved.

For a Fault Tolerant DA environment fail over to the Inactive DA and make the currently Active DA the standby Inactive DA. That will restart the REST web services when the Active DA shuts down and Inactive DA starts.

Additional Information

If this is a reproducible problem where the workaround doesn't work at all or it's only temporary, enable the following debug logging. Reproduce the issue and open a new support case. Reference this article and attach the resulting logging to the case for analysis. Ideally run the re.sh script and attach the resulting diagnostics logging package to the new support case. Instructions for running the re.sh script are found here.

1. Open the /opt/IMDataAggregator/apache-karaf-<version>/etc/org.ops4j.pax.logging.cfg file on the DA for editing.
2. At the bottom of each file add the following section.
   

   #
   # REST webservice endpoint debug
   # Added for support case 32978055
   # Add pound symbols to front of lines to disable.
   #
   log4j2.logger.AutomaticWebServiceManagerImpl.name = com.ca.im.odata.impl.AutomaticWebServiceManagerImpl
   log4j2.logger.AutomaticWebServiceManagerImpl.level = TRACE
   log4j2.logger.AutomaticWebServiceManagerImpl.additivity = false
   log4j2.logger.AutomaticWebServiceManagerImpl.appenderRef.sift.ref = sift
   log4j2.logger.AutoWebServiceHandlerImpl.name = com.ca.im.item.ws.impl.AutoWebServiceHandlerImpl
   log4j2.logger.AutoWebServiceHandlerImpl.level = TRACE
   log4j2.logger.AutoWebServiceHandlerImpl.additivity = false
   log4j2.logger.AutoWebServiceHandlerImpl.appenderRef.sift.ref = sift​​

3. Save the changes to the file.

No restarts are need. The logging begins once the file is edited and the changes are saved.