Endpoint Server error java.io.IOException: Too many open files

Products

• Data Loss Prevention Endpoint Prevent

Issue/Introduction

The Endpoint Server consumes around 100% CPU.

One of the following errors appear repeatedly in the Aggregator log files:

Apr 4, 2010 1:16:18 PM com.vontu.communication.transport.TCPAcceptOperation select
INFO: caught exception during accept
java.io.IOException: Too many open files
 at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
 at sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source)
 at com.vontu.communication.transport.TCPAcceptOperation.select(TCPAcceptOperation.java:42)
 at com.vontu.communication.transport.PlainAcceptOperation.select(PlainAcceptOperation.java:36)
 at com.vontu.communication.transport.EndpointAcceptOperation.invokeAcceptOp(EndpointAcceptOperation.java:189)
 at com.vontu.communication.transport.EndpointAcceptOperation.select(EndpointAcceptOperation.java:54)
 at com.vontu.communication.transport.AcceptWrapperOperation.select(AcceptWrapperOperation.java:94)
 at com.vontu.communication.transport.ChannelManager.dispatch(ChannelManager.java:385)
 at com.vontu.communication.transport.ChannelManager.run(ChannelManager.java:103)
 at com.vontu.communication.transport.VirtualChannelManager.run(VirtualChannelManager.java:11)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)


Mar 6, 2023 4:33:40 PM org.jboss.netty.channel.socket.nio.AbstractNioSelector
WARNING: Failed to accept a connection.
java.io.IOException: Too many open files
    at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:421)
    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:249)
    at org.jboss.netty.channel.socket.nio.NioServerBoss.process(NioServerBoss.java:100)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
    at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750)

Cause

The DLP service user has exceeded the limit on open files and is no longer allowed to open new files or connections.

Resolution

The Java exception is related to a high number of open files (which also applies to sockets on Linux). The default limit on Linux is set to 1024.

Use the following workaround:

On Linux modify the following file:
/etc/security/limits.conf 
in order to configure ulimit for each user. For example:

protect soft nofile 4096
protect hard nofile 63536

Please note:

1) The parameter protect soft nofile has to be greater than the number of endpoint agents
2) It is required to restart the Endpoint detection server service in order to apply the changes. In some cases you may need to reboot the Linux server, not just restart the service
3) If a xx-SymantecDLPDetectionServer.conf file exists in /etc/security/limit.d, set the "protect hard nofile" in that file to 65536
4) If ServerCommunicator.CONNECT_POLLING_INTERVAL_SECONDS.int is set too low can cause this error also.