Tuning For Threads On Linux

Products

VMware Tanzu Gemfire

Issue/Introduction

GemFire utilizes a large number of threads to deliver high throughput and low latency. As the volume of operations on the GemFire server increases, the number of threads required also rises. This can lead to resource limits that prevent the creation of additional threads.

On Linux, threads and processes are treated similarly, the limits that restrict the number of processes will also apply to the threads those processes generate. Additionally, threads require memory and other system resources. This guide will help you understand and adjust the relevant limits to enable the OS to support more threads.

The user process limit can be determined using the command “ulimit -u”.

Environment

This article applies to all supported VMware GemFire versions running on Linux-based environments.

Cause

As GemFire scales, the increasing number of threads may encounter OS-imposed resource limits, which can prevent the creation of new threads. The primary constraints include:

nproc: Limits the number of processes and threads a user can run simultaneously.
kernel.pid_max: Determines the maximum number of process IDs (PIDs) the OS can use concurrently.
vm.max_map_count: Restricts the number of memory mappings a process can have at one time.

Resolution

To ensure GemFire can create the necessary number of threads, the following OS-level limits should be adjusted:

Number of Threads GemFire Can Create (Max Ceiling):

Server-side thread creation is determined by the "max-threads" parameter, which is set when starting the server. The recommended value for this is the default setting of zero.

Case 1: (When max-threads is zero)

Num threads = (ClientConnections + function exec pool size) * (number of peers * 2)

Case 2: (When max-threads is non-zero)

Num threads = (maxThreads + function execution pool size) * (number of peers * 2)

The "Num threads" value calculated here represents the maximum ceiling based on the worst-case scenario. This ensures that the OS is configured to create the necessary number of threads if required.

Following are some of the OS-Level Limits:

nproc:

The “nproc” limit specifies the maximum number of processes and threads a user can run simultaneously. This limit should only be modified for the user running the GemFire server process.

- Check the current limit:
```
ulimit -u
```
- Modify the limit by updating /etc/security/limits.conf (or /etc/security/limits.conf.d/ or another file depending on your Linux distributions).
  
  Ensure both the soft and hard limits are set to the same value to maintain consistency and prevent potential issues.
  
  Sample /etc/security/limits.conf:
```
* soft nproc -1
* hard nproc -1
```

"*" indicates all users. "-1" unlimited.

If GemFire is the only service on the host, setting the limit to “unlimited” may be a suitable option.

For a specific limit, set “Num Threads” calculated above.

If this limit is reached the Thread::start() method will throw an java.lang.OutOfMemoryError with a message “unable to create new native thread”.

kernel.pid_max:

The kernel-wide “kernel.pid_max” limit determines the maximum number of process IDs (PIDs) that can be used concurrently across the entire OS. Since threads also use PIDs, this limit may need to be increased to support a large number of threads. This limit controls the size of the kernel's PID bitmap, so it should not be set too high without careful consideration.

Since this limit applies system-wide, it's important to consider all processes and threads on the host, not just those associated with GemFire.

- To check the current value:
```
sysctl -a | grep pid_max
```
- To set a new value:
```
sysctl -w kernel.pid_max=<new_value>
```
- To persist the change, edit /etc/sysctl.conf and add the line
```
kernel.pid_max=<new_value>
```
- A good starting point is:
```
Num Threads + AllOtherThreads
```

(where AllOtherThreads is the number of threads other processes are using on the host).

If this limit is reached the Thread::start() method will throw an java.lang.OutOfMemoryError with a message “unable to create new native thread”.

vm.max_map_count (Maximum memory mappings per process)

The kernel-wide vm.max_map_count limit controls the maximum number of memory mappings a process can have concurrently. This limit is applied on a per-process basis but can only be configured at the kernel level.

- To check the current value:
```
sysctl vm.max_map_count
```
- To set a new value:
```
sysctl -w vm.max_map_count=<new_value>
```
- To persist the change, update /etc/sysctl.conf, /etc/sysctl.d/, or another method specific to your Linux distribution.
```
vm.max_map_count=<new_value>
```
- A recommended formula for calculation:
```
Num Threads * 2 + 1000
```
- Apply changes using:
```
sudo sysctl -p
```

The value can be determined by: “Num Threads * 2 + 1000” (Where 2 is how many maps Java uses per thread, and 1000 accounts for other maps Java will make).

If this limit is reached, the JVM will exit with an "hs_err" crash report that may contain a stack similar to the following:

Stack: [<memory range>], sp=<stack pointer>, free space=<size>

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)

V [libjvm.so+<offset>] VMError::report_and_die()+<offset>


V  [libjvm.so+<offset>]  report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*)+<offset>


V  [libjvm.so+<offset>]  os::pd_commit_memory(char*, unsigned long, bool)+<offset>


V  [libjvm.so+<offset>]  os::commit_memory(char*, unsigned long, bool)+<offset>


V  [libjvm.so+<offset>]  JavaThread::create_stack_guard_pages()+<offset>

V  [libjvm.so+<offset>]  JavaThread::run()+<offset>

V  [libjvm.so+<offset>]  java_start(Thread*)+<offset>

C [libpthread.so.0+<offset>] start_thread+<offset>

It may also contain the error message:

Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.

Additional Information

If any of these limits are reached, the JVM may throw errors such as:

java.lang.OutOfMemoryError: unable to create new native thread
Native memory allocation (mmap) failed to map X bytes for committing reserved memory.
JVM crash with a hs_err report showing memory allocation failures.