ALERT: Some images may not load properly within the Knowledge Base Article. If you see a broken image, please right-click and select 'Open image in a new tab'. We apologize for this inconvenience.

MOM EM Crashing due to CLOSE_WAIT connections

book

Article ID: 222757

calendar_today

Updated On:

Products

APM

Issue/Introduction

The MOM start crashing. We found there are several  CLOSE_WAIT connections and it is increasing. When the limit is reached at MOM, the MOM stop working. The only way to resolve this issue to recycle MOM. The issue occurs again when this condition is meet. This is occurring in both PROD and NON-PROD environments.  

Cause

It seems like issue is cause by older agents (EPAgent and Java Agents) (some from APM release 9.6) that are reporting metrics to this APM cluster.

Environment

Release : 10.7.0

Component : Introscope

Resolution

For old EPAgent, stop the process.

For old WAS Agent, try disable Agent for now. If it is not feasible to do that, block the IP from MOM side

In IntroscopeEnterpriseManager.properties

isengard.server.banned.client.ipaddresses=<',' separated IPs>

Example:

isengard.server.banned.client.ipaddresses=10.10x.2x0.20,10.1x0.2x0.30

Additional Information

WIP