EM Java processes terminate for running server out of memory
Article ID: 203343
Updated On:
Products
Issue/Introduction
Having issue where the Collectors EM will terminate for running out of Memory.
All Collectors have been configured with the following:
lax.nl.java.option.additional=-Xms40960m -Xmx40960m -Djava.awt.headless=true -Dmail.mime.charset=UTF-8 -Dorg.owasp.esapi.resources=./config/esapi -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./logs/ -Xss512k -Djava.io.tmpdir=./work
If you look at the Introscope EM "G1 Old Gen" history, the Heap is never over 18GB and you see GC run to reduce size
When you look at TOP or FREE output on the Linux server, you see the EM Java process is using 44GB of Resident Mem though.
Need help to determine proper setup for these EM Java process so they continue to run and don't crash.
It’s not causing a HeapDump, the process is just terminating. There’s nothing in the logs that I can see, I will upload them.
Environment
Release : 10.7.0
Component : APM Agents
Resolution
VMware ballooning is a feature that leverages the balloon driver (vmmemctl) included in the VMware tools installed in the guest OS to release memory, in order to give it back to the host in case of contention. Note that, if the installed VMware tools are not running in the guest, ballooning will not be available for the virtual machine.
In order to force the guest to release memory pages, the balloon driver will simulate a memory contention scenario in the guest by increasing memory pressure on the OS (Inflating), which will make it uses its own native reclamation methods and release the least valuable pages.
And if the resources are really tight, it will force the guest OS to swap pages to its virtual disk. The fact that the guest OS itself decides which pages to swap is a good thing as it will select the ones that will have the least impact on the system instead of random pages like the host would do if it forced the VM to swap to its page file (.vswp), which could very well be important and heavily accessed pages, seriously hindering performances… Once the reclamation is done, the balloon driver will stop and give the pages back to the host (deflate).
It is possible to limit the amount of memory the balloon driver can reclaim with the advanced setting sched.mem.maxmemctl.
The mysteriously consumed memory may be due to VMware ballooning inflating.
Additional Information
Helpful to gather :
cat /proc/meminfo
cat /proc/sys/vm/overcommit_memory
cat /proc/sys/vm/overcommit_ratio
Feedback
