Troubleshooting KahaDB Log Growth and Disk Exhaustion in CA SOI 4.3

Products

• CA Service Operations Insight (SOI)

Issue/Introduction

In CA Service Operations Insight (SOI) 4.3, the ActiveMQ service may experience rapid KahaDB log file accumulation, leading to disk space exhaustion. This often manifests as an "Usage Manager Store is Full" error or ActiveMQ service failure. This article provides steps to resolve log accumulation and investigate the underlying cause.

Symptoms

• Rapid growth of db-*.log files in the kahadb directory.
• "Usage Manager Store is Full" error in log files.
• ActiveMQ service failure or instability.
• WARN | Transport Connection to: ... failed: Software caused connection abort in activemq.log.

Environment

• CA Service Operations Insight (SOI) 4.3
• Decoupled ActiveMQ deployment

Cause

Network-level instability or forcibly closed connections between the SOI Application Server and the ActiveMQ host can prevent the broker from properly cleaning up journal files, leading to disk space exhaustion.

Resolution

1. Stop the ActiveMQ Service: Stop the SOI MQ service on the dedicated ActiveMQ server.

2. Backup and Reset KahaDB: Move the existing kahadb folder to a backup directory to allow for a clean restart. mv <ActiveMQ_Install_Path>/data/kahadb <ActiveMQ_Install_Path>/data/kahadb.bak

3. Start the ActiveMQ Service: Start the SOI MQ service. The broker will recreate a fresh kahadb directory upon startup.

4. Verify Stability: Monitor the directory for further log growth. If the issue recurs, investigate firewall configurations and network stability between the SOI Application Server and the ActiveMQ host.

Additional Information

• For persistent connectivity issues, review the network traffic logs between the SOI Application Server and the ActiveMQ host.
• Contact Broadcom Support if the issue persists after these steps: Contact Broadcom Support.