Spectrum Alarms disappeared after failover.

Products

• Spectrum
• Network Observability

Issue/Introduction

There was a maintenance window in which the primary SpectroSERVER was stopped. We found the OneClick client showed the connection as down, instead of switched.  The secondary SS crashed, so we initialized and reloaded the database and started the secondary SS.  The alarms in OneClick disappeared for this SS.

Environment

Release : 10.4.1

Component : Spectrum Core / SpectroSERVER

Cause

When the SSdb is reloaded, the alarm table is reset.  The reason for this is the fault tolerant alarm service works to keep the primary and secondary alarms in sync.

Resolution

The secondary SS crashed unknowingly.
The primary was brought down.
The secondary then had the SSdb reloaded, which wipes out the alarms.
The secondary SS was restarted which updates OC with a "blank" alarm table.
The primary SS then starts, syncs to the secondary, and pulls the alarms from the secondary. It diffs the previous alarms, and sees a difference so tries to clear them.

This will only happen when the secondary is initialized/reloaded AND the primary is brought down, AND the secondary is started before the primary SS. It will not happen if the primary stays up or is brought up before the secondary. It will also not happen if the secondary db is not reloaded (ie - if you were to just stop it for patches).

Going forward if you have the secondaries crash and you bring the primaries down, bring the primary back up first. Fix the secondary, start it and let it run for 30 minutes or so to sync the alarms, then you can stop the primary.