Spectrum general SNMP trap troubleshooting

Products

• CA Spectrum
• DX NetOps

Issue/Introduction

We are sending traps from one of 3rd party application but traps are not showing in Spectrum console.

 

Environment

Release : 20.2

Component : Spectrum Core / SpectroSERVER

Cause

This document outlines some methods to troubleshoot traps received in Spectrum

Resolution

This is a basic troubleshooting diagram for traps. NOTE: if this is SNMPv3 traps for SNMPv3 Device, make sure there is a SNMPv3 profile associated with this device to properly decrypt the traps. 

 

Basic Trap Flow: 

 

1. First collect packet capture to verify the trap is landed on the SS server. 

- use tcpdump or wireshark and investigate the packets to confirm the device is sending the correct trap and has landed on the server on port 162

- if no traps are seen on port 162, check firewall rules

 

2. verify trap received on VNM model when count is increasing as traps come in:

NOTE: If the incoming trap is type SNMPv3, make sure there is a matching SNMPv3 profile in Spectrum to decrypt the trap. Else, the trap will be dropped. 

 

3. verify trap "source ip" is modeled in Spectrum. 

- check VNM model events for 0x10802 events with same trap source IP

- the device must be modeled in Spectrum to properly process traps and events. If the device is not modeled, run Discovery on the Source IP to model the device, then test traps again.

- if there are NO 0x10802 events for the source IP are showing, but VNM shows trap count increase (and not dropped the trap) then enable Alert Manager debug on VNM model > Dynamic Debugging

 

- Alert Manager debug outputs to $SPECROOT/SS/VNM.OUT

NOTE: Alert Manager debug is *very* noisy debug - take care to only enable this while reproducing the traps, and be sure to disable this debug once you are done troubleshooting, to avoid log bloat

 

4. If the device is modeled, then check device events for the trap. Check for generic 0x10801 events. 

- if 0x10801 events exist, then the problem is there is no AlertMap entry to map the trap to an Event

 

5. Make sure to use MIB Tools (or manually edit the files if you have the knowledge)  to verify there exists a proper AlertMap and EventDisp entry for the trap OID and Event Code

- check <SPECROOT>/custom/Events for these files

- verify CsPcause and CsEvFormat files exist for this event as well

Additional Information

NOTE: the TrapTroubleshooting HTML has been uploaded to this KB but is saved as a "DOC" file - this NEEDs to be changed to a "ZIP" file in order to be accessed.