Java Application gets Out of Memory exit code 137 due to MALLOC_ARENA_MAX

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:

Your Java application is getting error Exited with status 137 (out of memory). This behavior indicates application container memory has been exceeded (NOT the JVM hitting OOM).

The environment has large size Diego cells (for example, R4.large with 16 CPU cores).

Java application is under high load with JVM near heap memory limit for continuous periods of time.

Error Message

2018-01-29T19:52:07.20-0500 [API/0] OUT App instance exited with guid 6ed5f4ac-3603-4fc7-a7ae-2b12b55183c7 payload: {"instance"=>"b66fa54b-ab36-4052-7762-0f05", "index"=>0, "reason"=>"CRASHED", "exit_description"=>"APP/PROC/WEB: Exited with status 137 (out of memory)", "crash_count"=>5, "crash_timestamp"=>1517273525487748928, "version"=>"8eecb7dd-d9e7-4c18-aadc-e12e2d20e493"}

Environment

Cause

There is a Java buildpack glibc environment variable called MALLOC_ARENA_MAX, which sets the maximum number of arenas that can be allocated. This is set to unlimited by default in Java buildpack below version 4.9.

An arena is a scratchpad that glibc maintains to return smaller blocks to the requestor. In addition to the main arena, glibc malloc will allocate additional arenas during thread contention. The reason for the creation of arenas is to improve the performance of multithreaded processes.

Java applications running on large-sized Diego cells may encounter OutOfMemory as a result of new arenas being created. This happens when a heavy load (for instance when Java garbage collection occurs) and there is thread contention between the CPU cores resulting in many arenas which exceed the container memory limit. 

Resolution

MALLOC_ARENA_MAX needs to be set to 2 in order to prevent arena memory from exceeding container memory limit.

You can set MALLOC_ARENA_MAX on your application by the command:

cf set-env <app-name> MALLOC_ARENA_MAX 2

You can set a running-environment variable to apply this variable to all application in your org:

 cf set-running-environment-variable-group '{"MALLOC_ARENA_MAX":"2"}'

This issue will be fixed in java buildpack 4.9 and higher versions.