MOM EM Crashing due to CLOSE_WAIT connections

Products

• APM

Issue/Introduction

The MOM start crashing. We found there are several  CLOSE_WAIT connections and it is increasing. When the limit is reached at MOM, the MOM stop working. The only way to resolve this issue to recycle MOM. The issue occurs again when this condition is meet. This is occurring in both PROD and NON-PROD environments.  

Environment

Release : 10.7.0

Component : Introscope

Cause

It seems like issue is cause by older agents (EPAgent and Java Agents) (some from APM release 9.6) that are reporting metrics to this APM cluster.

Resolution

For old EPAgent, stop the process.

For old WAS Agent, try disable Agent for now. If it is not feasible to do that, block the IP from MOM side

In IntroscopeEnterpriseManager.properties

isengard.server.banned.client.ipaddresses=<',' separated IPs>

Example:

isengard.server.banned.client.ipaddresses=10.10x.2x0.20,10.1x0.2x0.30

Additional Information

WIP