Java Garbage Collection runs out of memory and cell CPU hits 100% usage in VMware vCloud Director 5.5.x
Article ID: 325482
Updated On:
Products
VMware Cloud Director
Issue/Introduction
Symptoms:
- Java Garbage Collection (GC) fails on VMware vCloud Director(vCD) cell due to insufficient heap memory.
- In the vCD debug log, you see entries similar to:
<YYYY-MM-DD>T<time> | WARN | pool-jetty-111012 | BatchTask | Error executing backend call. |java.lang.OutOfMemoryError: GC overhead limit exceeded</time>. - vCD cell CPU runs at 100% usage.
Note: This log excerpt is an example. Date, time, and environmental variables may vary depending on your environment.
- In the vCD debug log, you see entries similar to:
<YYYY-MM-DD>T<time> </time> | DEBUG | OSGI Delegator - listener dispatcher | EventHelper | Dispatcher service is unavailable. Queuing event: Event [id=<EVENT_UUID>, timestamp=XXXXXX, type=com/vmware/vcloud/event/cell/modify, serviceNamespace=com.vmware.vcloud, properties={...}]
Cause
The Java Virtual Machine (JVM) present in vCloud allocates memory for variables automatically whenever a new object is created.
When the variable is no longer referenced by any other piece of running code, the memory is marked for reclaim.
When a certain threshold of system memory is filled up, Java will trigger a process (Garbage Collector) to reclaim all those marked memory locations.
Depending on the configuration of the vCloud Director Cell and the current load, the Cell may not have enough resources to successfully run a successful Garbage Collection (GC).
In some circumstances, the scheduled purging of the audit_event data in the vCloud Database may have an adverse impact on the Garbage Collection as it would require resources from the JVM to be able to do so.
When the variable is no longer referenced by any other piece of running code, the memory is marked for reclaim.
When a certain threshold of system memory is filled up, Java will trigger a process (Garbage Collector) to reclaim all those marked memory locations.
Depending on the configuration of the vCloud Director Cell and the current load, the Cell may not have enough resources to successfully run a successful Garbage Collection (GC).
In some circumstances, the scheduled purging of the audit_event data in the vCloud Database may have an adverse impact on the Garbage Collection as it would require resources from the JVM to be able to do so.
Resolution
This issue is resolved in vCloud Director 5.5.3, available at VMware Downloads. For more information, see the vCloud Director 5.5.3 Release Notes.
If you are on a version of vCloud Director later than the version above, please verify if you are experiencing issues with the audit_event data clean up the activities.
For more information see Manually Cleaning Audit_Event Information from the vCloud Director Database.
Additional Information
Feedback
Yes
No
Powered by
