Largemessage filling VNA drive space repeatedly

Products

• CA Virtual Network Assurance

Issue/Introduction

We keep running into this problem every 2-3 days: https://knowledge.broadcom.com/external/article/224909/largemessage-filling-vna-drive-space.html

WARN  (Camel (broker-plugin-context) thread #0 - JmsConsumer[InventoryUpdateQueue])  [org.apache.camel.component.jms.DefaultJmsMessageListenerContainer] DefaultMessageListenerContainer 865  Setup of JMS message listener invoker failed for destination 'InventoryUpdateQueue' - trying to recover. Cause: readerIndex(4) + length(1) exceeds writerIndex(4): UnpooledDuplicatedByteBuf(ridx: 4, widx: 4, cap: 4, unwrapped: UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeHeapByteBuf(ridx: 4, widx: 4, cap: 4))

INFO  (Camel (broker-plugin-context) thread #0 - JmsConsumer[InventoryUpdateQueue])  [org.apache.camel.component.jms.DefaultJmsMessageListenerContainer] DefaultMessageListenerContainer 915  Successfully refreshed JMS Connection

Environment

Release : 20.2

Component :

Cause

The activemq cache is corrupted, due to running out of disk space.

Resolution

stop wildfly

remove the cached messages from activeMQ ( command below )
This will back the files up - we should be able to just delete them, but we can just move them first, and then delete them once we get VNA up and running after the restart.

mv /opt/CA/VNA/wildfly/standalone/data/activemq/journal/*.amq /tmp/backup

start wildfly