Portal contacts Anomaly Detector host after Data Source is deleted

Products

• Network Observability
• CA Performance Management

Issue/Introduction

The DX NetOps Performance Management Portal web server is trying to contact an Anomaly Detector (AD) Data Source (DS) host.

The DS being contacted was already deleted. There is no AD DS listed in the Manage Data Sources page accessed via Administration->Data Sources->Data Sources.

In the Event Manager EMService.log (default location is /opt/CA/PerformanceCenter/EM/logs) we see this error message every few minutes.

ERROR | pool-5-thread-7354       | 2024-02-21 12:37:33,071 | com.ca.im.portal.em.ds.EventProducerPoller                       
      | PollEventProducers
javax.xml.ws.WebServiceException: Could not send Message.
...
Caused by: java.net.UnknownHostException: UnknownHostException invoking http://<AD_DS_HostName>:80/AnomalyDetector/DataSourceWS.asmx: <AD_DS_HostName>
...
Caused by: java.net.UnknownHostException: <AD_DS_HostName>
...

Environment

All supported DX NetOps Performance Management releases

Cause

When the AD DS was deleted it was removed from the Portal netqosportal DB. For some reason, it failed to be properly removed from the Event Manager (EM) database.

This took place a long time before the issue was found. No logging remained available for review from the time of the initial deletion attempt.

The issue isn't reproducible in labs.

Resolution

To resolve this issue complete these steps.

1. Log into the Portal web UI.
2. Go to Administration->Data Sources->Data Sources for the Manage Data Source page.
3. Monitor the last sync time for both the Event Manager (EM) Data Source (DS) and the Global Sync last run time.
   
   • The Global Sync details only update with a browser tab level refresh. Use that to refresh everything.
   • Note the last time both completed updates.
4. Open a terminal to the Portal hosts CLI. Run the following MySQL command.
   • /opt/CA/MySql/bin/mysql -uroot -p -e "UPDATE em.data_sources SET UpdatedOn=0 WHERE SourceID=6;"
   • Change the path to yours if different than the default.
   • Enter the MySql DB password when prompted.
   • This triggers steps to complete the deletion of that data source in the software.
5. After the MySql command is run monitor the Manage Data Sources page updates to the EM DS and Global Sync last run times.

After the EM DS and Global Sync times update, monitor the EM service logs. We should see the errors clear up and the EM service stop trying to communicate with the old DS.