When running availability reports in Spectrum Report Manager there are ongoing outages from a long time ago

Products

• CA Spectrum
• DX NetOps

Issue/Introduction

When running availability reports in Spectrum Report Manager (SRM) there are ongoing outages from a long time ago

Within the Spectrum Report Manager (SRM) ongoing outages indicate that an outage started but has not yet ended. Ongoing outages that have existed for long periods of time (i.e. greater than 1 day) can indicate that the clear event for that outage was lost or not processed.

These are most commonly lost due to the Archive Manager being down for extended periods. When Archive Manager is not available, the SpectroSERVER is responsible for storing events locally in memory. By default the SpectroSERVER will keep 10,000 or 20,000 events. In this age of large devices, this can be used up in a matter of minutes. Once the limit is hit, the SpectroSERVER will start to purge the earlier events increasing the chance your up event will be lost.

Another way this could happen is if the database was copied or transferred from another SRM host or SpectroSERVER host. If the device was down when the backup was made and came up before the backup was applied to the new system - you will have an ongoing outage as a result.

Environment

Release: Any
Component: SPCOCK

Resolution

NOTE: In the following MySql commands, replace <PASSWD> with the root password for your DX NetOps Spectrum version.

There are a couple solutions to fix the actual ongoing outages.

The first, and easiest is to utilize the Report Manager Outage Editor. You can end outages via the UI, and then set them to exempt. However, this method is not the easiest if you have many of these outages.

In this case, MySQL will need to be used to end as many as we can manually - then the outage editor can be used to clean up. The following instructions will; connect to mysql, find outages that are ongoing for greater than 1 day (if you have legitimate outages then adjust the query to reflect this) and end them with todays timestamp, and exempt those outages so they do not count against overall availability of the report.

1. Navigate to $SPECROOT/mysql/bin
2. Type: ./mysql --defaults-file=../my-spectrum.cnf -uroot -p<PASSWD> reporting
3. Type this query:
   
   Update modeloutage set end_time=now() and outage_type=3 where datediff(now(), start_time)>1 and end_time is null;

It is highly recommended to make a backup copy of the modeloutage table before making ANY changes.

To make a backup, do the following:

1. Navigate to $SPECROOT/mysql/bin
2. Type: mysqldump --defaults-file=../my-spectrum.cnf -uroot -p<PASSWD> reporting modeloutage > modeloutage_dump.sql

This will create a backup in the $SPECROOT/mysql/bin directory.

You should now utilize the Outage Editor to do any clean up of outages that we did not get with the above update query.