Alarms are not updating in the nas setup window, IM or OC

Products

• DX Unified Infrastructure Management (Nimsoft / UIM)
• CA Unified Infrastructure Management On-Premise (Nimsoft / UIM)
• CA Unified Infrastructure Management SaaS (Nimsoft / UIM)

Issue/Introduction

We are unable to see any live alerts in nas.  Alarms are forwarding from collector HUBs and reaching up to the central (core HUB) in a GET queue but no new alarms are displaying in the nas or IM alarm subconsole or the Operator Console (OC).

Here are some errors seen in the _nas.log file :

Mar 14 14:37:28:386 [11924] 2 nas: NameService terminated...
Mar 14 14:37:28:387 [6576] 3 nas: NameService destroyed...
Mar 14 14:37:28:387 [6576] 3 nas: Destroying Database service...
Mar 14 14:37:28:396 [12256] 2 nas: Database service terminated...
Mar 14 14:37:28:396 [6576] 3 nas: Database service destroyed...
Mar 14 14:37:28:397 [6576] 0 nas: NAS Terminated.
Mar 14 14:37:28:398 [6576] 4 nas: CONNECT: 00000233532D5010(1780) ##.##.#.##/57896->#.##.#.#/48000
Mar 14 14:37:28:398 [6576] 5 nas: SREQUEST: port_unregister ->##.##.#.##/48000
Mar 14 14:37:28:398 [6576] 5 nas: RREPLY: status=OK(0) <-##.##.#.##/48000  h=37 d=0 fd=1780
Mar 14 14:37:28:398 [6576] 5 nas: SREQUEST: _close ->##.##.#.##/48000
Mar 14 14:37:28:398 [6576] 2 nas: nimEnd

• Yesterday at about midday, 11:54, new alarms STOPPED being displayed in the nas, IM and OC
• Asked customer if any changes were made at that time but they were not aware of any changes
• The only alarms displayed were from yesterday before the incident
• Tested alarm generation and reception by sending a test alarm from the nas Setup window as well as the test_alarm probe utility command in the controller but still no alarms displayed.

Environment

• Release: UIM 20.3
• nas 9.32

Cause

• multiple issues, including nas 9.32, nas table corruption, and a host of nas preprocessing (exclude) rules.

Resolution

• Upgraded nas 9.32 to 9.32HF1
• nas.log showed nas table corruption due to duplicate columns, so we deactivated nas and AE, renamed the nas local nas tables database.db and transactionlog.db, (dropped) the backend tables, due to nas alarm table corruptions:

DROP TABLE NAS_VERSION;
DROP TABLE NAS_ALARMS;
DROP TABLE NAS_TRANSACTION_SUMMARY;
DROP TABLE NAS_TRANSACTION_LOG;
DROP TABLE NAS_NOTES;
DROP TABLE NAS_ALARM_NOTE;

• Activated nas and AE
  
  
  
  
  
  
• Noticed that the customer had numerous nas pre-processing exclude rules, so deactivated multiple preprocessing rules to see if new alarms started arriving and they did.
  • Recommended that the customer reconcile ALL of the preprocessing rules, to make sure nothing is/remains too broad/interfering with alarms being displayed, e.g., filters/regexes, host lists.
• Overall, multiple issues with nas setup and cfg, table corruption, and too many exclude rules
• Flow of nas alarms restored.