Collector showing "configuration state" as "Not Connected"

Products

• CA Performance Management - Usage and Administration
• DX NetOps

Issue/Introduction

Would like to troubleshoot this to understand why some collectors go into this state after a bit. So far my resolution would be to bounce activemq. But doing this flushes any queued data. This is not desirable. How do we troubleshoot this issue to better understand "what" is wrong?

Environment

Release : 21.2

Component : PM Polling

Cause

In the DA /opt/IMDataAggregator/broker/apache-activemq-X.XX.X/data/activemq.log we can see the following WARN messages for 2 of the DC's. This is causing the configuration state to show "Not Connected" in the GUI. 

2022-05-11 10:03:03,345 | WARN  | Usage(default:memory:queue://DIP-poll.responses.irep-capc-poller-01.company.com:20a7c54d-20c3-4095-a8d3-48bad3e2899f:memory) percentUsage=100%, usage=209826887, limit=209715200, percentUsageMinDelta=1%;Parent:Usage(default:memory) percentUsage=6%, usage=570942759, limit=8737574093, percentUsageMinDelta=1%: Usage Manager Memory Limit reached. Producer (ID:capc-da-02.company.com-39610-1648027519953-3:41:1:1) stopped to prevent flooding queue://DIP-poll.responses.irep-capc-poller-01.company.com:20a7c54d-20c3-4095-a8d3-48bad3e2899f.

2022-05-11 10:04:32,288 | WARN  | Usage(default:memory:queue://DIP-poll.responses.irep-capc-poller-02.company.com:f478d29f-f782-4a59-a248-1bafd66ddf17:memory) percentUsage=100%, usage=209952227, limit=209715200, percentUsageMinDelta=1%;Parent:Usage(default:memory) percentUsage=6%, usage=571364821, limit=8737574093, percentUsageMinDelta=1%: Usage Manager Memory Limit reached. Producer (ID:austx-capc-da-02.chtrse.com-39610-1648027519953-3:5:2:1) stopped to prevent flooding queue://DIP-poll.responses.irep-capc-poller-02.company.com:f478d29f-f782-4a59-a248-1bafd66ddf17.

Resolution

On the DA  purge the queues mentioned in the WARN message in the Data Aggregator activemq.log file: 
/opt/IMDataAggregator/scripts/purgeOneQueue DIP-poll.responses.irep-capc-poller-01.company.com:20a7c54d-20c3-4095-a8d3-48bad3e2899f
/opt/IMDataAggregator/scripts/purgeOneQueue DIP-poll.responses.irep-capc-poller-02.company.com:f478d29f-f782-4a59-a248-1bafd66ddf17

Then restart activemq on the Data Collector(s) 
systemctl activemq stop 
systemctl activemq start