NAS probe on HA hub turns red after resetting database files on primary NAS

Products

• DX Unified Infrastructure Management (Nimsoft / UIM)

Issue/Introduction

After a severity-1 incident in which the primary hub NAS stopped processing alarms, which was resolved by resetting the NAS local database files, the NAS probe on the HA hub turned red.

The log on the HA NAS probe shows only a "max" restarts error:

Max. restarts reached for probe 'nas' (command = nas.exe)

Environment

DX UIM 23.4.x
NAS probe (any version) deployed on HA hub
database.db and transactionlog.db on primary NAS recently deleted/reset

Cause

Database mismatch between primary and HA NAS.

When the database.db and transactionlog.db on the primary NAS are reset, this clears all current alarms and resets the NAS to a fresh state in which all alarms are seen as new alarms from that point forward.

The HA hub still has a record of the old alarm states and the replication of the incorrect alarm states between primary and secondary HA NAS can cause database corruption.

Resolution

1. Stop the NAS probe on the primary hub and the HA hub.
2. Copy the database.db and transactionlog.db files from the Primary hub to the HA hub's NAS folder
3. start the NAS probes

For future incidents, if the database/transactionlog files are reset/cleared on the Primary NAS, they should always be reset on the HA HAS at the same time to avoid this issue.