Updating devices to utilize SNMPv3 instead of SNMPv2c.
After changing the devices SNMP Profile to the correct SNMPv3 based profile, the device switches back to its old still valid SNMPv2c profile.
If the old SNMPv2c Profile is deleted, and the device is set to the correct SNMPv3 based profile, it first shows Green/Normal for Status in the Data Aggregator Inventory. After one poll cycle it changes to Contact Lost for the status.
If deleting the device and trying to rediscover it as new, it is only found as a Pingable.
This is all despite confirming that the correct credentials return a valid response using SNMP request tools at the command line of the Data Collector that will manage and poll these devices.
The IP for the problem device is found in the following ERROR message in the managing Data Collectors karaf.log file. It will list each IP showing the same EngineID in use for an SNMPv3 device. In this message example there are four devices using the same EngineID.
2020-02-12 12:43:52,863 | ERROR | Response Pool.2 | MPv3Proxy | m.ca.im.dm.snmp.snmp4j.MPv3Proxy 978 | 196 - com.ca.im.data-collection-manager.snmp - 3.7.1.RELEASE-384 | | Duplicate EngineID detected: 80:00:00:09:03:00:00:00:00:00:00:00, IP(s) associated [<IP_Addr_1>/161, <IP_Addr_2>/161, <IP_Addr_3>/161, <IP_Addr_4>/161]
Default path to the Data Collector karaf.log is:
All supported Performance Management releases
Devices are configured in a way that duplicates the engine ID between devices.
Every SNMPv3 device must have a unique engine ID not shared by other devices.
This is described in RFC 5343 found here: https://tools.ietf.org/html/rfc5343
To validate is this is the cause we can use information from the DC snmpSession data. To access it we'll use different methods depending on the Performance Management release involved.
In the output from that URL search for the entries that show known SNMPv3 engine ID values and IP Addresses that have reported them. Where there is more than one IP Address against a given SNMPv3 engine ID, we have multiple devices reporting the same engine ID.
Sample section showing three devices reporting the same engine ID.
EngineBoots: 8, EngineTime: 2999050
MULTIPLE ADDRESSES:[22.214.171.124/161, 126.96.36.199/161, 188.8.131.52/161]
Note that the cache is not cleared until the DC is restarted so if the engineID has changed on the device you may see still see the older entry.
Reconfigure affected devices to ensure they have unique SNMPv3 engine ID values.