How can I recover a corrupted \ActiveMQ\data\kahadb ?

Products

• CA Virtual Privilege Manager
• CA Privileged Identity Management Endpoint (PIM)
• CA Privileged Access Manager (PAM)

Issue/Introduction

Due to some problems with disk space, we purged the ActiveMQ queues by deleting some files from inside the \ActiveMQ\data\kahadb

Following this operation, any attempt at starting ActiveMQ fails and we get the following error in the wrapper.log file from ActiveMQ:

INFO | jvm 1 | 2018/12/12 10:19:03 | INFO | Apache Camel 2.16.2 (CamelContext: camel) started in 10.672 seconds 
INFO | jvm 1 | 2018/12/12 10:19:03 | INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[E:\ActiveMQ\bin\..\data\kahadb] 
INFO | jvm 1 | 2018/12/12 10:19:04 | INFO | KahaDB is version 6 
INFO | jvm 1 | 2018/12/12 10:19:04 | INFO | Recovering from the journal @2964:33290807 
INFO | jvm 1 | 2018/12/12 10:19:04 | INFO | Recovery replayed 1 operations from the journal in 0.062 seconds. 
INFO | jvm 1 | 2018/12/12 10:19:04 | WARN | Some journal files are missing: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, ...] 
INFO | jvm 1 | 2018/12/12 10:19:04 | INFO | [0:queue/audit] dropped: ID:xxxxxx-49187-1520868776470-0:1:1:1:1 at corrupt location: 1:9595760 
INFO | jvm 1 | 2018/12/12 10:19:04 | INFO | [0:queue/audit] dropped: ID:xxxxxx-49187-1520868776470-0:1:1:1:2 at corrupt location: 1:10731710 
INFO | jvm 1 | 2018/12/12 10:19:07 | INFO | Detected missing/corrupt journal files. Dropped 29230 messages from the index in 2.906 seconds. 
INFO | jvm 1 | 2018/12/12 10:19:07 | INFO | PListStore:[E:\ActiveMQ\bin\..\data\xxxxxx\tmp_storage] started 
INFO | jvm 1 | 2018/12/12 10:19:07 | INFO | Installing Discarding Dead Letter Queue broker plugin[dropAll=true; dropTemporaryTopics=true; dropTemporaryQueues=true; dropOnly=ac_server_to_server_broadcast; reportInterval=60000] 
INFO | jvm 1 | 2018/12/12 10:19:08 | INFO | Apache ActiveMQ 5.13.1 (xxxxxx, ID:wbogvmapp162-49798-1544627947706-0:1) is starting 
INFO | jvm 1 | 2018/12/12 10:19:08 | INFO | Listening for connections at: nio+ssl://wbogvmapp162:61616?maximumConnections=1000&needClientAuth=false&transport.enabledProtocols=TLSv1.2&wireFormat.maxFrameSize=104857600 

And the service stops. How can I fix it ?

Environment

Privileged Identity Manager 14.0 and PAM Server Control 14.0/14.1 Enterprise Management

Resolution

The problem is due to corruption of the kahabdb files. The following procedure will recreate them;

1) Stop ActiveMQ service
2) In the ActiveMQ installation directory rename kahadb folder to kahadb_old
3) In the same directory rename tmp to tmp_old
4) Start Active MQ service.

That should recreate the queues

Note: The renamed kahadb folders can be deleted (kahadb_old and tmp_old) after all is up and running fine.