Contact lost to secondary alarms happen consistently - 0x10c0e event
Article ID: 131917
Updated On:
Products
Issue/Introduction
The alarm "Contact lost to secondary SpectroSERVER on host <hostname> port 0xbeef at precedence 20 from primary SpectroSERVER on host <hostname> port 0xbeef." is happening consistently daily.
Environment
Release: Any release
Component: SPCCSS
Cause
Resolution
The secondary server is shutting down during an online backup. This is due to how we load the database on the secondary after an online backup occurs.
This is functioning as designed up to 10.3.2.
https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/spectrum/22-2/administrating/distributed-spectroserver-administration/about-spectroserver-fault-tolerance.html
Generate an Alarm If the Secondary SpectroSERVER Is Not Restarted
When a primary SpectroSERVER synchronizes its database with the secondary SpectroSERVER, a Contact Lost to Secondary Server (0x00010c0e) event and alarm are generated. The secondary SpectroSERVER has been brought down to load the new database from the primary SpectroSERVER.
You can set up a rule to process this alarm so that the alarm is generated only if the secondary SpectroSERVER is not restarted.
The EventPair rule lets you specify that a new event is generated if the Contact Lost to Secondary Server event occurs and a Contact Established to Secondary Server (0x00010c0f) event does not follow within a specified time period. You can then specify that this new event creates an event and an alarm indicating that the secondary SpectroSERVER is still down.
An example of how to implement the EventPair rule:
1. Create or edit the existing $SPECROOT/custom/Events/EventDisp file and add the following two lines:
0x10c0e E 50 R CA.EventPair, "0x10c0f", "0xfff00000 -:-", 300
0xfff00000 E 0 A 2,0x10c0e
NOTE: 0xfff00000 is a custom event code. You can use any custom event code available in your environment.
If the 0x10c0e event is generated and it is not followed by the 0x10c0f event within 5 minutes (300 seconds), then the 0xfff00000 event is generated. The 0xfff00000 event will assert the Major alarm Cause Code 0x10c0e.
2. Click on the "Update Event Configuration" button under the SpectroSERVER Control subview of the VNM model to reload the changes made in the $SPECROOT/custom/Events/EventDisp file.
3. Launch the Event Configuration Editor utility.
4. Copy the Event Message of the 0x10c0e event code.
5. And paste to the 0xfff00000 event code. (Note that 0xfff00000 is just an example, you may use any other available event code)
6. Save the changes in Event Configuration Editor.
7. Perform an OnLineBackup. You should no longer see the 0x10c0e event code with a Major alarm asserted.
8. You should see a 0x10c0e event code without a Major alarm.
9. If the 0x10c0e event code is not followed by the 0x10c0f event code within 5 minutes (300 seconds), then the 0xfff00000 event code is generated with a Major alarm and the Cause Code alarm is 0x10c0e.
So you may need to adjust the specified time period (in my example 5 minutes - 300 seconds) depending on how long the secondary take to be up and running.
Review the events and see how long took to generate the 0x10c0f event after generating the 0x10c0e event.
Also, review the following section of the Spectrum guide if you have multiple OneClick servers:
If you are running multiple OneClick web servers, copy the folders containing the event format files and the probable cause files to the other OneClick servers in your distributed environment. Copy the following folders:
- $SPECROOT/custom/Events/CsEvFormat
- $SPECROOT/custom/Events/CsPCause
And then click on "Update Event and Alarm Files" on the OneClick Administration page to reload the changes.
Also, copy the contents of these same directories to a custom directory on all of the SpectroSERVERs in your environment in the following circumstances:
- Use the command-line interface (CLI) commands showalarms or showevents.
- Use DX NetOps Spectrum Alarm Notification Manager (SANM).
Attachments
Feedback
