It seems that when we are getting SNMP traps to the snmptd probe it sometimes is creating the alarms that it should and other times it is not.
I am getting the alarm in the trap monitor and there is a profile for it to create an alarm from the message only no alarm is created.
Other times it will create an alarm, or sometimes it won't clear the alarm. However each time we are receiving the traps.
snmptd probe alarm is not clearing in the PROD environment.
- hub queue setup
Fixed hub nas queues
Current hub queues (as) setup on the Secondary and Primary were related to the snmptd issue.
We set the Secondary hub nas queues back to the way they should be 'out-of-the-box' in both PROD and TEST (Secondary hub) servers, (eliminated ATTACH and GET queues), and instead just used nas replication (Unidirectional) from the Secondary hub TO the Primary but...
We also needed to make sure that any legacy artifacts of ems-related queues didn't re-surface...and it did so when the customer was still able to recreate an old issue where the alarm_enrichment Subject erroneously switches to alarm1. This issue was related to previous UIM/ems versions and ems-related queues.
Therefore we proceeded to:
1. Disable the event_manager queue on the Primary as that should not be enabled by default in UIM 20.4
2. I examined all the existing hub queues, and as it turns out, although the customer was able to reproduce the snmptd clear alarm issue in Test and Prod, I fixed the issue of alarm_enrichment queue which was erroneously changing its default Subject of alarm to alarm1 in both environments.
Turns out it was actually happening in Test because even though you could see 'alarm' change to 'alarm1' and then the customer purposely changed it back to 'alarm' in the Queue Tab, the 'alarm Id' was still displayed in the hub Status window as value 'alarm1.'
Shown below, is the way it should be-> alarm and alarm, for Subject/Queue and Id, not alarm and alarm1:
You MUST change that back to alarm manually. So we changed the nas probe's 'enrichment_subject' setting back to alarm (from alarm1) and then restarted the nas.
Then the issue no longer returned/reoccurred during subsequent tests. The customer was able to confirm the fix, by making a change to a nas AO Profile and restarting it again.
Then once the original default queues were re-configured and the nas replication was setup between the Secondary hub ->the Primary, the snmptd probe trap alarms successfully cleared.
So in Summary:
a) hub queues needed to be reset to their out-of-the-box defaults for nas and alarm_enrichment
b) nas replication was then enabled and configured on the Secondary hub as Unidirectional (Secondary hub TO Primary)
c) nas probe's 'enrichment_subject' setting in the raw config was still set to alarm1 instead of alarm so we changed it to alarm.