search cancel

Collector showing "configuration state" as "Not Connected"

book

Article ID: 241492

calendar_today

Updated On:

Products

CA Performance Management - Usage and Administration DX NetOps

Issue/Introduction

Would like to troubleshoot this to understand why some collectors go into this state after a bit. So far my resolution would be to bounce activemq. But doing this flushes any queued data. This is not desirable. How do we troubleshoot this issue to better understand "what" is wrong?

Cause

In the DA /opt/IMDataAggregator/broker/apache-activemq-X.XX.X/data/activemq.log we can see the following WARN messages for 2 of the DC's. This is causing the configuration state to show "Not Connected" in the GUI. 

2022-05-11 10:03:03,345 | WARN  | Usage(default:memory:queue://DIP-poll.responses.irep-capc-poller-01.company.com:20a7c54d-20c3-4095-a8d3-48bad3e2899f:memory) percentUsage=100%, usage=209826887, limit=209715200, percentUsageMinDelta=1%;Parent:Usage(default:memory) percentUsage=6%, usage=570942759, limit=8737574093, percentUsageMinDelta=1%: Usage Manager Memory Limit reached. Producer (ID:capc-da-02.company.com-39610-1648027519953-3:41:1:1) stopped to prevent flooding queue://DIP-poll.responses.irep-capc-poller-01.company.com:20a7c54d-20c3-4095-a8d3-48bad3e2899f.

2022-05-11 10:04:32,288 | WARN  | Usage(default:memory:queue://DIP-poll.responses.irep-capc-poller-02.company.com:f478d29f-f782-4a59-a248-1bafd66ddf17:memory) percentUsage=100%, usage=209952227, limit=209715200, percentUsageMinDelta=1%;Parent:Usage(default:memory) percentUsage=6%, usage=571364821, limit=8737574093, percentUsageMinDelta=1%: Usage Manager Memory Limit reached. Producer (ID:austx-capc-da-02.chtrse.com-39610-1648027519953-3:5:2:1) stopped to prevent flooding queue://DIP-poll.responses.irep-capc-poller-02.company.com:f478d29f-f782-4a59-a248-1bafd66ddf17.

Environment

Release : 21.2

Component : PM Polling

Resolution

On the DA  purge the queues mentioned in the WARN message in the Data Aggregator activemq.log file: 
/opt/IMDataAggregator/scripts/purgeOneQueue DIP-poll.responses.irep-capc-poller-01.company.com:20a7c54d-20c3-4095-a8d3-48bad3e2899f
/opt/IMDataAggregator/scripts/purgeOneQueue DIP-poll.responses.irep-capc-poller-02.company.com:f478d29f-f782-4a59-a248-1bafd66ddf17

Then restart activemq on the Data Collector(s) 
systemctl activemq stop 
systemctl activemq start