The Application Crashes with the message, "cgroup out of memory" in Pivotal Cloud Foundry

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:

The application is crashing.

cf app <app-name> reports high memory utilization for application.

Diego cell reports kernel message: "cgroup out of memory."

Error Message:

/var/log/messages on container Diego cell:
<3>2017-04-26T19:12:36.397506+00:00 10.28.34.27 kernel [job=diego_cell index=2] [2437091.184092] Memory cgroup out of memory: 
Kill process 20316 (java) score 1016 or sacrifice child

Environment

Cause

The application has exhausted its allocated memory.

“Memory cgroup out of memory” is a message coming from the kernel when it kills something because of a memory cgroup restriction, such as those we place on containerized CF workload. If applications are running with high memory utilization then a memory spike without enough headroom could result in the kernel killing the containerized process.

Resolution

The memory for the application needs to be increased or the application needs to be re-written to utilize less memory.

1. Check the current application memory usage

cf app <app-name>
state since CPU memory disk details
#0 running 2017-04-26 01:32:46 PM 0.0% 958.8M of 1G 252.3M of 2G

2. Increase the memory allocated to the application

cf scale <app-name> -m <higher memory level>

This will require an application restart

3. Afterwards, you can monitor memory utilization with the following command

cf app <app-name>
state since CPU memory disk details
#0 running 2017-04-26 01:32:46 PM 0.0% 1087.8M of 2G 252.3M of 2G