Upgraded database and alarm_enrichment queue is down

Products

• DX Unified Infrastructure Management (Nimsoft / UIM)

Issue/Introduction

Upgrade of Oracle database from 11g to 19c.
hub shows alarm_enrichment queue backed up with 25k+ alarms and queue displays as yellow.

Jun 29 15:22:05:937 [main, alarm_enrichment] ****************[ Starting ]**************** 
Jun 29 15:22:05:938 [main, alarm_enrichment] 9.31 
Jun 29 15:22:06:047 [main, alarm_enrichment] Nas: Starting: Alarm Enrichment Process 
Jun 29 15:22:06:083 [main, alarm_enrichment] CmdbJdbcDataSource: GenericCmdbJdbcDataSource intializing with config path enrichment-source/cmdbs/dev_id 
Jun 29 15:22:06:087 [main, alarm_enrichment] CmdbJdbcDataSource: connecting to database 
Jun 29 15:22:10:863 [main, alarm_enrichment] CmdbJdbcDataSource: ORA-01033: ORACLE initialization or shutdown in progress 
Jun 29 15:22:10:863 [main, alarm_enrichment] (1) error, Failed to connect to database: ORA-01033: ORACLE initialization or shutdown in progress 
Jun 29 15:22:10:865 [main, alarm_enrichment] (1) error, Failed to connect to database: ORA-01033: ORACLE initialization or shutdown in progress 
    at com.nimsoft.probe.service.enrichment.cmdb.CmdbJdbcDataSource.connectToDatabase(CmdbJdbcDataSource.java:113) 
    at com.nimsoft.probe.service.enrichment.cmdb.CmdbJdbcDataSource.initialize(CmdbJdbcDataSource.java:57) 
    at com.nimsoft.probe.service.enrichment.cmdb.CmdbCacheEnrichmentSource.createDataSourceHandlerForConfigEntry(CmdbCacheEnrichmentSource.java:191) 
    at com.nimsoft.probe.service.enrichment.cmdb.CmdbCacheEnrichmentSource.initialize(CmdbCacheEnrichmentSource.java:122) 
    at com.nimsoft.probe.service.enrichment.cmdb.CmdbCacheEnrichmentSource.readCmdbSection(CmdbCacheEnrichmentSource.java:66) 
    at com.nimsoft.probe.service.nas.alarmenrichment.AlarmEnrichmentService.loadEnrichmentSources(AlarmEnrichmentService.java:96) 
    at com.nimsoft.probe.service.nas.alarmenrichment.AlarmEnrichmentService.<init>(AlarmEnrichmentService.java:40) 
    at com.nimsoft.probe.service.nas.Nas.postLoginInitialization(Nas.java:466) 
    at com.nimsoft.probe.service.nas.Nas.login(Nas.java:418) 
    at com.nimsoft.nimbus.NimProbeBase.nimAttachProbe(NimProbeBase.java:303) 
    at com.nimsoft.nimbus.NimProbe.initSubscribe(NimProbe.java:572) 
    at com.nimsoft.nimbus.NimProbe.doForever(NimProbe.java:416) 
    at com.nimsoft.probe.service.nas.Nas.run(Nas.java:611) 
Jun 29 15:22:12:015 [140449853417280] Controller: Max. restarts reached for probe 'alarm_enrichment' (command = <startup java>)

Environment

• Release: 20.3
• alarm_enrichment v9.31

Cause

• Nas was trying to connect to the old database

Resolution

• Updated nas.cfg to point to new db.