MOM Has too many open files after 20 minutes /socket exception

Products

CA Application Performance Management (APM / Wily / Introscope)

Issue/Introduction

Logs repeating .

Works 10-15 minutes . Then gets this . And port 5001 drop connections

[DEBUG] [Incoming Connection Listener] [Manager.IncomingConnectionListener] Acquiring connection ticket

[DEBUG] [Incoming Connection Listener] [Manager.IncomingConnectionListener] Calling server socket accept()

[ERROR] [Incoming Connection Listener] [Manager.IncomingConnectionListener] SocketException has been thrown, probably authentication is failed

[DEBUG] [Incoming Connection Listener] [Manager.IncomingConnectionListener] SocketException has been thrown, probably authentication is failed java.net.SocketException: Too many open files (Accept failed) at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:535) at java.net.ServerSocket.implAccept(ServerSocket.java:545) at java.net.ServerSocket.accept(ServerSocket.java:513) at com.wily.isengard.postofficehu

Environment

Release : 10.7.0

Component : Introscope

Resolution

An internet search shows this is all documented:

Search terms search: apm white list agents site:broadcom.com

Agent side (Java Agent):

https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/application-performance-management/10-7/implementing-agents/java-agent/configure-java-monitoring/configure-java-agent.html

Q10: Can you recommend best practices to implement a whitelist/blacklist approach for agents, e.g. through loadbalancing.xml?

Answer:

Load balancing is not a good way to implement whitelist for agents as a security measure.
You can use network firewall of the operating system (e.g. iptables on Linux) to restrict access to TCP port 5001 (and TLS port 5443 if enabled)

For LB information and other options how to block agents you can refer to the below 2 links:

Is there a way to block some agents connections to a Enterprise Manager?
https://knowledge.broadcom.com/external/article/187111

https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/application-performance-management/10-7/administrating/configure-enterprise-manager/configure-mom-agent-load-balancing.html

https://knowledge.broadcom.com/external/article/187111
here is an Enterprise Manager property that will allow to ban the IP addresses from agents. Please follow below steps:

1.- Open the IntroscopeEnterpriseManager.properties file for the Enterprise Manager you would like to block agents
2.- Add the following property:
isengard.server.banned.client.ipaddresses=<Agent1_IP_Address,<Agent2_IP_Address>,<Agent3_IP_Address>

eg:
Agent1 IP Address: 11.157.207.6
Agent2 IP Address: 11.157.207.12
Agent3 IP Address: 11.157.207.13
isengard.server.banned.client.ipaddresses=11.157.207.6,11.157.207.12,11.157.207.13

3.- Save the file
4.- Restart the Enterprise Manager for the changes to take effect