Data Collectors Stuck in 'Catching Up' due to Full /DASharedRepo NFS Volume
Article ID: 452709
Updated On:
Products
Issue/Introduction
All Data Collectors in the NetOps Portal show status: CATCHING_UP.
DA karaf.log shows multi-million message backlogs for DC IDs in DC-POLLING-STATE log lines.
DA ActiveMQ Poll Response Queue (PRQ) shows Memory% = 100.
DA activemq.out log entries include:
java.io.IOException: No space left on device
java.io.IOException: Invalid location size
ERROR | Checkpoint failed — No space left on device
Environment
Product: DX NetOps Performance Management
Component: Data Aggregator (DA), Data Collector (DC)
Deployment: NFS-mounted /DASharedRepo for ActiveMQ KahaDB storage.
Cause
The /DASharedRepo NFS volume reached 100% capacity, preventing the ActiveMQ KahaDB from writing or checkpointing its journal. This leads to index corruption.
Resolution
[!CAUTION] Performing a "clean" restart of the DA will clear the unpersisted Poll Response Queue. Ensure disk space is restored before proceeding to allow a healthy rebuild of the message index.
Step 1: Restore Disk Space
Verify the /DASharedRepo mount usage: df -h /DASharedRepo.
Engage storage or Linux administration teams to expand the volume or reclaim space.
Check for and archive old certification sets (e.g., DASharedRepo/DA/certifications/) or stale diagnostic bundles to free up space.
Step 2: Perform a Clean Restart of the Data Aggregator
Once space is available, the corrupted ActiveMQ index must be cleared:
Log in to the Data Aggregator server as the root or sudo user.
Navigate to the scripts directory:
bash
cd /opt/IMDataAggregator/scripts/
Run the following cleanup sequence:
bash
./dadaemon maintenance
./dadaemon clean
./dadaemon activate
Step 3: Verify Data Flow
Monitor the DA karaf.log and verify that the queueSize for each DC is steadily decreasing.
Confirm that the DC Polling Status in the NetOps Portal transitions from CATCHING_UP to NORMAL or Ready.
Preventative Measures
Implement proactive monitoring and alerting for the /DASharedRepo disk usage thresholds.
Increase the Data Collector ActiveMQ broker heap (minimum 4GB recommended) to better handle backlogs during future DA outages.
Feedback
