Java Application Memory not Being Garbage Collected
Article ID: 297830
Updated On:
Products
Issue/Introduction
The memory of java application gradually increases over time. The below commands shows continually increase in memory utilization:
cf <app-name> running 2017-06-13 02:52:30 PM 36.8% 705.2M of 1G 137.2M of 1G.
Eventually, the application will reach OutOfMemory and application will be restarted.
Environment
Cause
An understanding of garbage collection in Java is necessary to troubleshoot continual increasing memory usage. This behavior could be caused by the memory leak in the application. There are also differences in memory management of Java buildpacks in PCF. Additionally, there are tools such as VisualVM that can be used to monitor and diagnosis memory utilization of apps.
Resolution
Upgrade to Java buildpack 4.x:
Upgrade Java buildpack to version 4.x which has enhancements to memory handling.
See: https://www.cloudfoundry.org/just-released-java-buildpack-4-0/
This includes:
- Improved JVM Memory calculation
- Histogram outputted to application log in event of crash
- Ability to capture heap dump for OutOfMemory events
Application memory monitoring:
VisualVM with Visual GC plugin can be used to monitor and diagnosis Java applications, see tutorial: http://www.oracle.com/webfolder/technetwork/tutorials/obe/java/gc01/index.html
The following KB article describes how to setup tunnel such that you can monitor PCF app remotely: https://discuss.pivotal.io/hc/en-us/articles/221330108-How-to-remotely-monitor-Java-applications-deployed-on-PCF-via-JMX
Visual VM can be used to trend memory utilization, monitor when garbage collection kicks in, and gather statistics related to memory and GC.
Memory Leak Identification:
Applications with continually increasing memory should be reviewed for memory leaks. The following article provides tips on identifying memory leaks: https://developers.redhat.com/blog/2014/08/14/find-fix-memory-leaks-java-application/
If developers compile apps in eclipse then it would point out potential memory leaks in code. Running Visual GC on these apps would give a lot of information as well as the ability to manually trigger GC. Developers could also try manually disable & enable parts of your code and observe memory usage.
Java Heap Dump Generation:
In order to generate a heap dump for java, you will need to install cf cli java plugin. This is necessary since heap dump generated on the container will be immediately removed when the container is torn down after application restart. The java plugin will allow heap dump to be outputted to stdout.
See link: https://github.com/SAP/cf-cli-java-plugin
Additionally, starting in 4.2 [https://github.com/cloudfoundry/java-buildpack/releases/tag/v4.2] there is added the ability to write a terminal heap dump to a volume service if one is bound [https://github.com/cloudfoundry/java-buildpack/blob/master/docs/jre-open_jdk_jre.md#jvmkill].
Feedback
