Smarts IP: How to simulate a DownOrFlapping or Unstable alert with a trap

Products

VMware Smart Assurance

Environment

VMware Smart Assurance - SMARTS

Resolution

This article explains how to generate a trap that will simulate DownOrFlapping alert or Unstable alert in Smarts IP.

Simulating the DownOrFlapping or Unstable alerts in Smarts IP can be done by sending linkDown traps to the Smarts IP domain manager. The linkDown trap is defined in the RFC standard by the following:

linkDown NOTIFICATION-TYPE
    OBJECTS { ifIndex, ifAdminStatus, ifOperStatus }
    STATUS current
    DESCRIPTION
            "A linkDown trap signifies that the SNMP entity, acting in
            an agent role, has detected that the ifOperStatus object for
            one of its communication links is about to enter the down
            state from some other state (but not from the notPresent
            state). This other state is indicated by the included value
            of ifOperStatus."
    ::= { snmpTraps 3 } = 1.3.6.1.6.3.1.1.5.3

                SNMPv2:

OID = .1.3.6.1.6.3.1.1.5.3
varbind 1 = ifIndex
varbind 2 = ifAdminStatus
varbind 3 = ifOperStatus

                snmpTraps ::= { snmpMIBObjects 5 }
                snmpMIBObjects ::= 1.3.6.1.6.3.1.1

With the generic trap code being 2 and the specific trap code being 0.

Example simulation
The following example simulation assumes you have a switch that has an IP address of 192.168.101.3 with a gigabit interface that has an ifIndex of 10110. The switch with IP address 192.168.101.3 has to be discovered in the Smarts IP domain for the Smarts software to accept the trap and alert on it.

To simulate DownOrFlapping and Unstable alerts in this environment, do the following:

Log into the Smarts IP domain and navigate to the Domain Manager Administration Console (DMAC) from the Topology Browser console:
In the DMAC open the Polling and Thresholds settings:
In the Polling and Thresholds window click on the Thresholds tab at the bottom left of the screen:
Expand the Interface Groups setting and then expand the 1 GB Ethernet setting:
By default the Interface/Port flapping setting is not enabled as can be seen in the right hand window labeled Available Settings:

Select Interface/Port Flapping and then click Add to move it to the enabled setting:
Once the setting has been moved to the right hand window lableled current settings click Apply at the bottom of the screen and then Reconfigure at the top left of the screen:
After clicking reconfigure expand the Settings for the 1 Gb Ethernet and select Interface/Port Flapping:
In the right hand window we can see that there are two settings:

LinkTrapThreshold- This is the number of linkDown traps that have to be received before Smarts will alert that there is a problem with the interface. This is set to 3 by default.
LinkTrapWindow - This is the number of minutes in which the trap must come in as a set to cause an alert. The default is 300 seconds.

With the above settings, three linkDown traps would have to come in within a 5-minute window of the first trap for an alert to be generated. How long the alert lasts is based on when the last linkDown trap comes in and the LinkTrapWindow. If only three linkDown traps come in, and generate a DownOrFlapping event and the link trap window is set to 300 seconds, then the alert will from smarts will clear 5 minutes after the third trap came in. If the link trap window is set to 3600 seconds (max allowed length) then the alert will clear after the last link down trap comes in.

The alert timer gets reset to the LinkTrapWindow time ever time a new trap comes in while the alert is still active.

Set the LinkTrapThreshold to 5 and the LinkTrapWindow to 3600, click Apply then Reconfigure as in step 6:
Now that the trap settings are complete it is time to build the trap that will be sent. First, some information needs to be gathered.
1. What port the Smarts IP software is listening to for traps: By default this port is 9000 and is set in the <base dir>/IP/smarts/conf/trapd/trapd.conf and is overridden by <basedir>/IP/smarts/local/conf/trapd/trapd.conf. The value in the trapd.conf file is the PORT setting. In our simulation the port is 9999 and not 9000.
2. The IP addres of the Smarts IP domain: In this case it is 10.241.215.36.
3. The IP address of the device sending the trap: In this case the swich IP address is 192.168.101.3.
4. The ifIndex of the interface that the trap is for: In this simulation that is ifIndex 10110.
5. The Interface Oper Status and the interface Admin Status: In this simulation they will both be 1, for up.
Building the trap using the definition for the link trap at the top and using the sm_snmp command to sent the trap we get the following:

./sm_snmp -d 10.241.215.36 -p 9999 trap 192.168.101.3 .1.3.6.1.6.3.1.1.5.3 2 0 555 .1.3.6.1.6.3.1.1.5.3.1.2 i 10110 .1.3.6.1.6.3.1.1.5.3.1.3 i 1 .1.3.6.1.6.3.1.1.5.3.1.4 i 1

Where

-d 10.241.215.36 = destination IP address for the trap. The Smarts IP domain host.
-p 9999 = The port the trap is sent to on the host.
trap - Instructis the sm_snmp program that it is sending a trap.
192.168.101.3 = The point of orogin of the trap.
.1.3.6.1.6.3.1.1.5.3 = Trap OID.
2 = Generic code for a link down trap.
0 = Specific code for a link down trap.
555 = Duration of the trap in seconds.
.1.3.6.1.6.3.1.1.5.3.1.2 i 10110 = First varbind of type integer(i) representing the ifIndex 10110.
.1.3.6.1.6.3.1.1.5.3.1.3 i 1 = Second varbind of type integer(i) representing ifAdminStatus 1 (up).
.1.3.6.1.6.3.1.1.5.3.1.4 i 1 = Third varbing of tupe integer(i) representing ifOperStatus 1 (up).
Send the five traps by executing the trap command above five times from the <basedir>/IP/smarts/bin directory. Root/administrative access may be required depending on host settings:

Once five of these traps are received a DownOrFlapping and Unstable alert should be present in the Smarts IP domain classic console, and possibly a Down alert that should clear shortly in favor of the Unstable alert:

Any SAM domain subscribed to the IP domain should show just an Unstable alert, or an Unstable alert and a cleared Down alert.

Additional Information

It is important to note that although a DownOrFlapping is generated in the SM-IP-AMPM-DownFlapping domain it is not generated in the SM-SAM-DownFlapping domain. This is because the SAM domains do not subscribe to DownOrFlapping alerts by default but instead subscribe to Down or Unstable alerts. This behavior can be changed but is not the subject of this article.

To subscribe to a DownOrFlapping event please review the following KB article: https://support.emc.com/kb/183055

Attachments

SimulatingDownOrFlap.docx get_app