Unique Alerts (based on traps) for servers in cluster

Products

Network Observability Spectrum

Issue/Introduction

We are sending traps from 2 servers for Service status on same OID.

We want alert for service down on one server and alert when same service up on other server.

How do I separate servers? as both servers are receiving same traps and I want only running alert for server A and only stopped alert for server B

Resolution

Section 1: Configuration Steps (SpectroSERVER & OneClick)

1. Identify Your Variables and Event Codes Before starting, define the custom event codes you will use and note your environment details.

Background Router Event: e.g., 0xfff00001
Result Event A (Server 1 Alarm): e.g., 0xfff00002
Result Event B (Server 2 Alarm): e.g., 0xfff00003

2. Update the AlertMap Map the incoming trap to the background event and extract the variables.

File: /opt/CA/Spectrum/custom/Events/AlertMap
Rule of Thumb: The Trap OID here must exactly match the OID string the SpectroSERVER receives. If you are unsure, temporarily enable "Alert Manager" Dynamic Debugging on the VNM model to see the exact OID string arriving.
Template:

Plaintext 

<EXACT_TRAP_OID>  0xfff00001  <VARBIND_1_OID>(1,0) <VARBIND_2_OID>(2,0)

(Example: 1.3.6.1.4.1.11536.3.6.6.6.4 0xfff00001 1.3.6.1.4.1.11536.2.5.1(1,0) 1.3.6.1.4.1.11536.2.5.2(2,0))

3. Update the EventDisp (The Logic) Evaluate the background event, check the device's IP Address (Attribute 0x12d7f), check the varbind payload, and explicitly trigger the alarms.

File: /opt/CA/Spectrum/custom/Events/EventDisp
Rule of Thumb: Use Regexp to avoid strict string-casting failures, and always use {C CURRENT_MODEL} for the target.
Template:

Plaintext 

0xfff00001 E 50 P " \
If( \
   And( \
      Regexp( ToString(ReadAttribute( {C CURRENT_MODEL}, {H 0x12d7f} )), {S \"<SERVER_1_IP>\"} ), \
      Regexp( ToString(GetEventVariable( {U 2} )), {S \"<STATUS_A>\"} ) \
   ), \
   CreateEventWithVariables( {C CURRENT_MODEL}, {H 0xfff00002}, GetEventVariableList() ), \
   If( \
      And( \
          Regexp( ToString(ReadAttribute( {C CURRENT_MODEL}, {H 0x12d7f} )), {S \"<SERVER_2_IP>\"} ), \
          Regexp( ToString(GetEventVariable( {U 2} )), {S \"<STATUS_B>\"} ) \
      ), \
      CreateEventWithVariables( {C CURRENT_MODEL}, {H 0xfff00003}, GetEventVariableList() ), \
      Nil() \
   ) \
)"

4. Reload SpectroSERVER Configuration

In OneClick, select the VNM model.
Go to Information > SpectroSERVER Control.
Click Update Event Configuration.

5. Configure Alarms & Titles in the Event Configuration Editor

In OneClick, go to Tools > Utilities > Event Configuration.
Click Create Event (green +) for 0xfff00001, 0xfff00002, and 0xfff00003.
For your Alarm events (0xfff00002 and 0xfff00003):
- Event Message: Enter the text you want displayed in the Events tab.
- Alarms Tab: Set the Severity (e.g., Major/Critical) and check "Separate alarms for each instance" (use discriminator 1 to separate by Service Name). Change the Alarm Title from "ALARM TITLE" to a meaningful message (e.g., "Server is Running").
Click File > Save All (This creates the UI format files!).

6. Reload OneClick Cache

Go to the OneClick web administration page: http://<OneClick_IP>/spectrum/admin/
Click Administration > Update Event and Alarm Files.
Click Reload.

Section 2: Testing Steps (Using the Simulation Script)

Create a bash script (trapsim.sh) to send SNMPv2c traps mimicking your servers.

Important Simulation Gotchas:

Varbind OIDs: If your AlertMap expects instance 0 (e.g., (1,0)), your script must append .0 to the varbind OIDs (e.g., <VARBIND_1_OID>.0). Otherwise, Spectrum ignores the payload, and your event variables will be blank.
Base Trap OID: The snmptrap utility might automatically append sub-identifiers to the Trap OID you provide. Always confirm the final OID string in the VNM.OUT debug trace to ensure your AlertMap matches what is actually being sent.

Test Script Template:

Bash 

#!/bin/bash

# --- CONFIGURATION ---
SS_IP_ADDRESS="<SPECTROSERVER_IP>"
COMMUNITY="<SNMP_COMMUNITY>"
BASE_TRAP_OID="<EXACT_BASE_TRAP_OID>"

send_running_trap() {
    echo "Sending trap for Condition A..."
    snmptrap -v 2c -c "$COMMUNITY" "$SS_IP_ADDRESS":162 '' $BASE_TRAP_OID \
        <VARBIND_1_OID>.0 s "Service Name Example" \
        <VARBIND_2_OID>.0 s "<STATUS_A>"
}

send_stopped_trap() {
    echo "Sending trap for Condition B..."
    snmptrap -v 2c -c "$COMMUNITY" "$SS_IP_ADDRESS":162 '' $BASE_TRAP_OID \
        <VARBIND_1_OID>.0 s "Service Name Example" \
        <VARBIND_2_OID>.0 s "<STATUS_B>"
}

# --- MAIN EXECUTIONS ---
# (Note: Use iptables SNAT or run this script directly from the target servers 
# to ensure the source IP maps correctly to the device models in Spectrum)

send_running_trap
sleep 2
send_stopped_trap

Verification: After running the script, navigate to the target device models in OneClick.

The Events tab will show your beautifully formatted custom messages with the extracted variables.
The Alarms tab will show the newly generated alarms with your custom Alarm Titles!