Configuring File Descriptors on Linux for GemFire Servers

Products

• VMware Tanzu Gemfire

Issue/Introduction

This article describes how GemFire uses file descriptors and how these should be configured on the Linux level for use on a GemFire server or peer host machine or VM.

This article is based on this archive VMWare article by Edin Zulich.

Environment

OS: Linux

Resolution

Open file descriptors (FD's) are an OS resource used for file handles, threads, and network connections (sockets). Given the distributed nature of GemFire deployments, it is often the case that GemFire servers have many socket connections open as they communicate with other servers in the same cluster, clients, or other GemFire clusters via WAN gateway connections. Usually, the default Linux limit for the number of open file descriptors is not sufficient for GemFire deployments. 

As a general guideline, we recommend setting the FD limit to 50000 or higher, depending on the estimated use. Linux, for example, uses roughly 1KB of RAM per one open file descriptor for bookkeeping. So, the memory overhead for 50000 open file descriptors is less than 50MB. Given such a low overhead, it is better to over-allocate than risk running out of file descriptors and bringing a production system to a stall.

One way to easily determine the FD usage in a running environment is to monitor the GemFire FD usage statistic. You can view this statistic using VSD. For more information see File Descriptor Issue. 

One other thing that complicates things here and should be taken into account is the fact that JVM itself can contribute to an increase of open file descriptors: the Oracle JVM uses internal per-thread selectors for blocking network I/O, and each one of those selectors uses three file descriptors. The JVM does not clean up these selectors and their file descriptors until garbage collection. Because of this, the number of these selectors and file descriptors can keep growing, and if GC does not happen often enough, the FD limit can be reached. The Oracle JVM uses sun.nio.ch.Util$SelectorWrapper$Closer objects to clean up these selectors and their file descriptors; if a heap histogram shows a large number of them, there will also be three times as many FD's in use, which can be verified using the command line tool lsof. Upon GC, the number of Closer objects and file descriptors will go down.

Controlling GemFire's use of FDs

Controlling GemFire's use of file descriptors may involve tuning JVM garbage collection and GemFire settings that control the use of sockets and threads:

1. Garbage collection on the servers: Use CMS garbage collector, and CMSInitiatingOccupancyFraction set to a level that ensures a regular and timely GC cycle.
2. Increase the socket-lease-time GemFire property on the servers (Default = 60000 ms), or set to 0 to disable timeout altogether.
3. On the clients, Increase the client pool idle-timeout (Default = 5000 ms), or even turn it off by setting it to -1.

Modifying the Limit of Open File Descriptors on Linux

Limits can be checked using ulimit:

ulimit -n -S

For a soft limit on open files per process, and:

ulimit -n -H

For hard limit on open files per process. Limits are configured in /etc/security/limits.conf, access to which requires superuser privileges.