Troubleshooting SNMPv3 in Spectrum
search cancel

Troubleshooting SNMPv3 in Spectrum

book

Article ID: 139839

calendar_today

Updated On:

Products

CA Spectrum DX NetOps

Issue/Introduction

We have an SNMPv3 device that is continuously losing contact with Spectrum.  We can reach this device every time with an snmpwalk even when Spectrum cannot communicate.

How can we troubleshoot this further?

SNMP query works from server CLI but device is not responding to discovery from the OneClick UI.

Attempting to update device to SNMPv3 in Spectrum. Device responds to original SNMPv2 credentials. Device fails IP discovery. ICMP responds but SNMPv3 does not.

Attempt to test through MIB Tool -> Contact Criteria seems to hang, never responds to either SNMPv2 nor SNMPv3.

From server command line, SNMPv3 responds using the sapwalk2 command and the same SNMPv3 credentials set in Spectrum.

Environment

All supported DX NetOps Spectrum releases

Cause

In SNMPv3 all Engine ID´s must be unique, from the Engine ID Spectrum will store the Engine Time which cannot exceed 150 seconds and the Engine Boots which can never decrease in value.

When another device on the network has the same Engine ID the Engine Time and Engine Boots will not match and this causes the problem.

Resolution

To determine if the is the problem take the following steps.

  1. Open a terminal to the server.
  2. Go to the $SPECROOT/vnmsh directory and run this in a bash shell to enter the Spectrum command prompt.
    • ./connect
  3. In the Spectrum command prompt run the following to dump the SNMPv3 cache stored by the running SpectroSERVER.
    • ./update action=0x10331 mh=<VNM_MH>
    • It writes output to the VNM.OUT file in the $SPECROOT/SS directory.
    • Replace <VNM_MH> with the Model_Handle Attribute value for the SS's VNM model found in the OneClick UI.
  4. Start a Wireshark trace (Windows) or tcpdump (Linux). Ideally filter it to capture only packets related to the IP of the problematic device.
    • Recommended tcpdump command for Linux. Often requires root user access to run the command.
      • tcpdump -w /tmp/troubleshoot.pcap -vv -A -T snmp -s 0 "(dst port 162) or (src port 161) or (dst port 161) and (host <SpectroSERVER_IP>)"
        • -w tells it where to write the output file and it's name. Change as needed.
        • Change dst (destination) and src (source) parts as needed. Defaults are shown.
        • Replace <SpectroSERVER_IP> with the SS hosts IP address.
    • In Windows run Wireshark and start the capture. Use this in the filter field to limit packets to only those of the IP of the problem device.
      • ip.addr == <IP_Address>
      • Replace <IP_Address> with the the IP of the problem device.
  5. In the Spectrum command prompt run the following to clear the SNMPv3 cache stored by the running SpectroSERVER.
    • ./update action=0x10333 mh=<VNM_MH>
    • Replace <VNM_MH> with the Model_Handle Attribute value for the SS's VNM model found in the OneClick UI.
  6. Reproduce the issue:
    • If Discovery from the OneClick UI fails reproduce that failure.
    • Device showing an incorrect Lost Contact state? Wait for a few poll cycles to pass. Review VNM.OUT logging for a cause.
  7. Once the device is discovered successfully or regains positive a contact status, examine the packet capture.
    • Compare it to the previous values from the dump of SNMPv3 cache

The Engine Boots cannot be a lower number and the Engine Time has to be less then 150 seconds difference. If either of these are present, then most likely there are devices with duplicate SNMPv3 EngineID values.

If you have any further doubts or questions, please provide the packet capture, VNM.OUT logs and the SNMPv3 cache dump to Support for further analysis. To analyze the data we'll need the SNMPv3 community string to decode the encrypted packets in the packet capture data.

When reviewing packet capture data if we see an SNMPv3 Report being sent from the device using OID 1.3.6.1.6.3.15.1.1.5.0 (wrong Digest) this indicates the credentials used are incorrect. If the credentials are correct ensure the default SNMPV3 settings for protocols match what is being sent from the device. See these values in the $SPECROOT/SS/.vnmrc file.

  • snmpv3_default_auth_protocol=<value>
  • snmpv3_default_priv_protocol=<value>

For example if these settings are set but the device is using SHA and 3DES then you have to prefix the community string between the password as follows in this sample.

  • #v3/P:SHA^authpassword:3DES^privpassword/userid

Additional Information

Powered by Wolken Software