Debugging Spectrum's SNMP Traffic

Products

• CA Spectrum

Issue/Introduction

Description:

Over the years a number of customers have asked the question: "What SNMP OIDs does Spectrum use to manage my network?" It turns out this simple question has a fairly complex answer that depends on:

• Vendor
• Device Type
• Firmware Revision
• Device Configuration
• Network Configuration
• Device Connections

To begin addressing this it is good to understand the scope of the question. CA Technology's Spectrum product has over 2600 MIBs imported into its database and application models for over 700 of these MIBs. So in practice it has been nearly impossible to tell customers in advance exactly which SNMP OIDs Spectrum will request in their network.

Since this question needs an answer, creative approaches need to be developed to meet this need. This tech bulletin outlines ways customers can answer this in their own environment.

Although using a Packet Analyzer to capture all traffic can potentially answer this question, its use has some limitations:

• Security policy in some environments does not permit running promiscuous packet captures
  
  
• The user does not have the ability to install additional software on the SpectroSERVER machine
  
  
• The Packet Analyzer captures both the request and the response possibly exposing sensitive data.
  
  
• In encrypted SNMPv3 communications the packet captures will not be usable.
  
  
• The output of the Packet Analyzer needs additional processing to extract needed information

For this reason an approach using only Spectrum is considered in this tech bulletin.

Solution:

The Tools

There are tools built into Spectrum which permit the run time tracing of SNMP packets which are issued by the SpectroSERVER. This has been a standard part of the product for since very early versions of the product. Additional "tools" described in this bulletin are needed to parse the SNMP debug output. These tools are simple UNIX commands and scripts which the user can run on trace data. The information produced by the SNMP Stack debugging contains volumes of data which is not relevant to answering the question above. The tools filter that out and produce more useful information

The commands (or scripts) recommend are standard on UNIX systems and can be obtained on the Windows platform as well. These were developed and tested with Cygwin utilities on a Windows 7 platform.

The Process

The process of capturing all OIDs used includes the following steps:

• Enable SNMP Communication Tracing
  
  
• Perform Discovery or other operations to be monitored
  
  
• Disable SNMP Communications Tracing
  
  
• Process the debug output generated

There are potentially two possible mechanisms to enable / disable SNMP Communications Tracing depending on which version of Spectrum is in use. All Spectrum versions support the CLI based action which is sent via the following command:

     update action=0x10314 mh=<VNM_MH> # to Enable Tracing
     update action=0x10315 mh=<VNM_MH> # to Disable Tracng

Another alternative for customers of Spectrum 9.2.2 (or later) is to use the One Click UI to enable and disable the tracing. This can be done using the following controls on the VNM Information View:

<Please see attached file for image>

The user should set the value of "SNMP Stack" to "Enabled" to begin the tracing process. After that the operation under test (Discovery, Modeling, Device Reconfiguration, NCM firmware capture, etc) should be performed

The following UNIX commands are used to parse the SNMP debugging output:

    egrep "^\ Device IP: | \ [0-9]: " VNM.OUT | sed "s/ Varbinds (.)://" > SnmpRequests.txt     
    egrep -v Device SnmpRequests.txt | sed "s/ .*://" | sort -u > SnmpOIDs.txt 

It is best to copy and paste these into a script to be executed since the space characters may not be obvious.

The Output

As mentioned the basic output of the SNMP Request tracing is extensive. This output should be contained in the file:

  $SPECROOT/SS/VNM.OUT 

The script (or individual commands) should be run in the $SPECROOT/SS directory and they produce two files:

SnmpRequests.txt - listing of all OIDs requested by device in time ordered sequence

SnmpOIDs.txt - a sorted list of all unique OIDs requested during the test

These text files have the following contents:

  SnmpRequests.txt       
        Device IP: 138.42.96.7 
           0: 1.3.6.1.4.1.9.9.48.1.1.1.5.2 
        Device IP: 138.42.96.6
          0: 1.3.6.1.2.1.4.20.1.1
          1: 1.3.6.1.2.1.4.20.1.2
          2: 1.3.6.1.2.1.4.20.1.3

   SnmpOIDs.txt       
          1.3.6.1.2.1.1.1.0 
          1.3.6.1.2.1.1.2.0
          1.3.6.1.2.1.1.3.0
          1.3.6.1.2.1.1.6.0
          1.3.6.1.2.1.2.1.0

Summary

Using Spectrum's built-in debugging and some scripting it should be possible to analyze SNMP requests for nearly any situation. Although the VNM.OUT may contain significant information, the tools and techniques presented should assist the user in extracting the desired information.

An important caveat of this work is that it is still not possible to tell customers completely in advance what OIDs Spectrum will use. This approach requires access to a live network with devices modeled in Spectrum. But the intent is that once this process is executed in the customer environment they will have the information for further use. This includes

• Debugging Device and Agent based issues
  
  
• Provide Debugging information to CA Tech Support
  
  
• Setting access controls for their network operations

 

Resolution

Later versions of Spectrum the SNMP Stack Debug can be turned on from the VNM Model -> Dynamic Debugging