Data Aggregator Shows Down After Backup and Restore

Products

• Network Observability
• CA Performance Management

Issue/Introduction

After executing a backup and restore script on the Data Aggregator, the DA displays a down status in the Portal system status view ․​​​​​​​‍

SYMPTOMS:

• Data Aggregator shows as down in the Portal system status ․
• ActiveMQ service fails to start or function correctly ․

IMPACT: Data Aggregator is unavailable and cannot process metrics ․

Environment

DX NetOps Data Aggregator :: All Supported Versions

Cause

CONTEXT: Occurs after restoring the entire /DASharedRepo/ directory from a backup while the Data Aggregator and ActiveMQ services are actively running ․

Resolution

STEPS:

1․ STOP SERVICES

Stop the Data Aggregator and ActiveMQ services on both nodes ․

EXPECTED: File locks on the shared repository are released․

2․ RENAME CORRUPT KAHADB

Path: /DASharedRepo/broker/kahadb

Rename the kahadb folder to kahadb․old ․

EXPECTED: The corrupted database directory is moved aside to allow recreation․

3․ START DATA AGGREGATOR and ACTIVEMQ SERVICE

COMMAND: ./dadaemin activate

VERIFY SUCCESS:

• Check the Portal system status view to confirm the Data Aggregator shows as Up ․

Additional Information

ROOT CAUSE: The activemq database is corrupted when restoring the /DASharedRepo/ directory from a backup while the Data Aggregator services are still running ․ On Fault Tolerant Data Aggregators, the kahadb is specifically stored in the shared /DASharedRepo/ path ․

PREVENTION: Do not backup and restore the entire /DASharedRepo/ directory ․ Scripts should only target specific configuration directories such as /DASharedRepo/certifications/custom and /DASharedRepo/certifications/extensions ․