The collector got disconnected from MOM

Products

• CA Application Performance Management Agent (APM / Wily / Introscope)
• CA Application Performance Management (APM / Wily / Introscope)
• INTROSCOPE
• DX Application Performance Management

Issue/Introduction

Some collectors got disconnected from the MOM and one collector disconnected  frequently yesterday. Today, there were errors in collector log like outgoing message queue is not moving. 

Environment

Application Performance Management 10.7

Cause

Overworked Cluster.

Resolution

What was done:
- Move Docker Agents to a dedicated collector
- Provide Sizing Guidance 500K Metrics per Collector (default)
- Remove 3 calculators
- Modified loadbalancing.xml
- transport.outgoingMessageQueueSize=20000 on one collector
- introscope.enterprisemanager.loadbalancing.threshold=20000 (default)

Additional Information

Possible Next steps

High Impact Items
- Clean up Smartstor Metrics
- Move MQ Agents to own cluster. (Be it Cloud or Physical)
- OR Collector Weighting 

 

Medium Impact Items
- Beef up Hardware/OS on Agents/EM.
- Increase

•  transport.outgoingMessageQueueSize=20000 (Remaining EMs. Workaround at Best)
• transport.override.isengard.high.concurrency.pool.min.size=15
• transport.override.isengard.high.concurrency.pool.max.size=15

- Remove unneeded MOM/Agent features

- Follow Best Admin Practices (Med)