Unable to discover device with SNMPv3 in DX NetOps Performance Management

Products

• CA Performance Management
• Network Observability

Issue/Introduction

Updating devices to utilize SNMPv3 instead of SNMPv2c.

After changing the devices SNMP Profile to the correct SNMPv3 based profile, the device switches back to its old still valid SNMPv2c profile.

If the old SNMPv2c Profile is deleted, and the device is set to the correct SNMPv3 based profile, it first shows Green/Normal for Status in the Data Aggregator Inventory. After one poll cycle it changes to Contact Lost for the status.

If deleting the device and trying to rediscover it as new, it is only found as a Pingable.

This is all despite confirming that the correct credentials return a valid response using SNMP request tools at the command line of the Data Collector that will manage and poll these devices.

The IP for the problem device is found in the following ERROR message in the managing Data Collectors karaf.log file. It will list each IP showing the same EngineID in use for an SNMPv3 device. In this message example there are four devices using the same EngineID.

2020-02-12 12:43:52,863 | ERROR | Response Pool.2  | MPv3Proxy                        | m.ca.im.dm.snmp.snmp4j.MPv3Proxy  978 | 196 - com.ca.im.data-collection-manager.snmp - 3.7.1.RELEASE-384 |  | Duplicate EngineID detected: 80:00:00:09:03:00:00:00:00:00:00:00, IP(s) associated [<IP_Addr_1>/161, <IP_Addr_2>/161, <IP_Addr_3>/161, <IP_Addr_4>/161]

Default path to the Data Collector karaf.log is:

/opt/IMDataCollector/apache-<version>/data/log

Environment

All supported Performance Management releases

Cause

Devices are configured in a way that duplicates the engine ID between devices.

Every SNMPv3 device must have a unique engine ID not shared by other devices.

This is described in RFC 5343 found here: https://tools.ietf.org/html/rfc5343

To validate if this is the cause we can use information from the DC snmpSession data. To access it we'll use different methods depending on the Performance Management release involved.

Due to security requirements, the snmpSession bean data is only visible via a curl command. It's run on the DC terminal CLI using the loopback address. Open a terminal to the DC and issue the following command as the root or install owner user:

• curl http://127.0.0.1:8681/debug/dispatcher/container/com.ca.im.data-collection-manager.snmp/bean/snmpSession

In the output from that URL search for the entries that show known SNMPv3 engine ID values and IP Addresses that have reported them. Where there is more than one IP Address against a given SNMPv3 engine ID, we have multiple devices reporting the same engine ID.

Sample section showing three devices reporting the same engine ID.

MPv3 engineIDs: 

81:43:45:0f:87:32:7b:23:c6:5c:4b:2e:66 : 

EngineBoots: 8, EngineTime: 2999050 

MULTIPLE ADDRESSES:[10.10.10.10/161, 10.11.11.11/161, 10.12.12.12/161]

Resolution

Reconfigure affected devices to ensure they have unique SNMPv3 engine ID values.

After reconfiguring the affected devices the SNMPv3 engineID data cached in the snmpSession bean doesn't clear on its own. To clear it restart the dcmd service on the affected DCs. After restart confirm the duplicates are no longer referenced. Once cleared and each device using a unique engineID value, the discovery should now be successful.