Alerts from Secondary Hub is not Sent to Primary Hub through Message Queues

Products

• DX Unified Infrastructure Management (Nimsoft / UIM)

Issue/Introduction

We have an ongoing Issue where, not all alerts from Secondary Hubs are being sent to Primary Hub. We have Get and Attach queues configured. Despite that, we have only few alerts being sent to Primary. Need to understand and fix about this misbehavior. 

origin of alarms comes from WXXXXXXXXSHU01

see alarms on that hub.

no backup/queued alarms on origin hub

no nas forwarding or replication configured.

WXXXXXXXXSCH01 - upstream hub

*nas 8.42 was installed on the hub where the alarms were being generated.*

no nas rules interfering from the local hub (origin hub)  since send of test alarms with most of the same message data made it to the upstream hub.

ntevl probe alarm:

SQL Events: SQLSERVERAGENT (208 - Job Engine): SQL Server Scheduled Job 'MXXXXXX-XX_SYSTEM_DB.Back Up Database (Transaction Log)' (0x49D417524262BB488083091FC5E9DFBD) - Status: Failed - Invoked on: 2020-09-23 10:45:00 - Message: The job failed.  The Job was invoked by Schedule 39 (MXXXXXX-XX_SYSTEM_DB.Back Up Database (Transaction Log)).  The last step to run was step 1 (Back Up Database (Transaction Log)).

Environment

Release : 20.1

Component : UIM - HUB

- nas v9.06

Resolution

- nas 8.42 - upstream hub is 9.06 - nas needs to be the same version.

subsystem ID of alarm-> 1.1.11.1.2
Source-> 10.3.65.30
Robot: mXXXXXX-oh
user tag1 global_apps
User Tag 2 -> wil_win_gxp_sql

- Upgraded nas on downstream hub to v9.06, then applied HF6.
https://support.broadcom.com/external/content/release-announcements/CA-Unified-Infrastructure-Management-Hotfix-Index/7233


- Sent test alarms and they were generated locally on WXXXXXXXXSHU01 and then also received by the upstream hub, WXXXXXXXXSCH01.

- Nevertheless, there are a significant number of nas preprocessing rules on the upstream hub which can have an adverse effect on message processing.

- Also the nas_transaction_log has over 9.2M rows and that should be decreased in the nis bridge, e.g., set it lower, from 90 down to 45.

- nas Transaction Log Tab settings for retention aren't too bad but normally to keep the database.db and transactionlog.db files from growing too large its good to keep those retention settings to a minimum. Plus you can always check alarms/alarm history in the backend NA _* tables.

Additional Information

Attached KB Article to follow for easing nas administration and maintenance and eliminating unexpected outcomes:

nas Best Practices, Tips and Techniques (for Large Environments)

https://knowledge.broadcom.com/external/article/192089/nas-best-practices-tips-and-techniques-f.html