The collector got disconnected from MOM

book

Article ID: 196073

calendar_today

Updated On:

Products

CA Application Performance Management Agent (APM / Wily / Introscope) CA Application Performance Management (APM / Wily / Introscope) INTROSCOPE DX Application Performance Management

Issue/Introduction

Some collectors got disconnected from the MOM and one collector disconnected  frequently yesterday. Today, there were errors in collector log like outgoing message queue is not moving. 

Cause

Overworked Cluster.

Environment

Release : 10.7.0

Component : APM Agents

Resolution

What was done:
- Move Docker Agents to a dedicated collector
- Provide Sizing Guidance 500K Metrics per Collector (default)
- Remove 3 calculators
- Modified loadbalancing.xml
- transport.outgoingMessageQueueSize=20000 on one collector
- introscope.enterprisemanager.loadbalancing.threshold=20000 (default)

Additional Information

Possible Next steps

High Impact Items
- Clean up Smartstor Metrics
- Move MQ Agents to own cluster. (Be it Cloud or Physical)
- OR Collector Weighting 
 
Medium Impact Items
- Beef up Hardware/OS on Agents/EM.
- Increase
  •  transport.outgoingMessageQueueSize=20000 (Remaining EMs. Workaround at Best)
  • transport.override.isengard.high.concurrency.pool.min.size=15
  • transport.override.isengard.high.concurrency.pool.max.size=15
- Remove unneeded MOM/Agent features
- Follow Best Admin Practices (Med)