How to determine Governance Memory requirements to avoid outofmemory errors
Article ID: 202248
Updated On:
Products
Issue/Introduction
How can we correctly size and set our Memory for Governance?
When running a campaign IG it is taking a long time and throws out of memory errors in the log files:
00:01:49,833 ERROR [stderr] (Thread-666 (HornetQ-client-global-threads-2042388150)) Exception in thread "Thread-666 (HornetQ-client-global-threads-2042388150)" java.lang.OutOfMemoryError: GC overhead limit exceeded
00:02:46,590 INFO [stdout] (pool-9-thread-1) 2020-10-26 00:02:46,590 [pool-9-thread-1] INFO com.workpoint.server.ServerProperties - ServerProperties.setProperty() invoked for property= calculated.db.offset.millis.WPDS, value=0
00:02:11,313 WARN [org.springframework.jms.listener.DefaultMessageListenerContainer] (businessFlowCampaignCreationExecutorContainer-3) Setup of JMS message listener invoker failed for destination 'HornetQQueue[fixAlertsExecutorQ]' - trying to recover. Cause: Could not commit JMS transaction; nested exception is javax.jms.JMSException: HQ119014: Timed out waiting for response when sending packet 43
00:06:33,081 SEVERE [com.sun.jersey.spi.container.ContainerResponse] (default task-40) The exception contained within MappableContainerException could not be mapped to a response, re-throwing to the HTTP container: java.lang.OutOfMemoryError: GC overhead limit exceeded
Environment
Release : 14.3
Component : GovernanceMinder(Role & Compliance Manager)
Cause
This is a Java Virtual Memory resource allocation and cache limit configuration issue which can be resolved by tuning the environment.
Resolution
Please make sure the servers are on the latest patch, then tune JVM and cache.
-Xmx Defines the maximum size of heap memory.
For example, for your overall heap -Xmx20g sets maximum memory to 20GB. This memory is assigned as needed.
We recommend, for a 64bit system, that you allocate approximately 3GB of cache memory (RAM) for every 1,000,000 elements allowed in the cache memory (3 * maxElementsInMemory).
Cache memory is defined by the number of elements (users, resources, roles, and so on)
You will need to perform a review of the Entity Browser Universe summary page to find out how many of each there are.
Each "model" (shown in second screen shot) only needs to be calculated if being used in a certification
From an internal LAB test environment:
Eurekify
Configuration Users:
972
Configuration Resources:
41
Roles:
4
PortalUniverse
Configuration Users:
141
Configuration Resources:
0
Roles:
0
identityMangerImportTest
Configuration Users:
175
Configuration Resources:
0
Roles:
2
importtestdata
Configuration Users:
69
Configuration Resources:
83
Roles:
97
Add all of these and multiply by 3 to get your Cache element requirements:
972+41+4+141+0+0+175+0+2+69+83+97=1584 * 3 = 4752
so for this lab environment the Cache Element setting in the \CA\RCM\Server\eurekify-jboss\standalone\deployments\eurekify.war\WEB-INF\classes\ehcache-sageDal.xml file does not need to be adjusted from its default of 1million:
<defaultCache maxElementsInMemory="1000000" eternal="true"
And would results in a JVM requirement of around 2.5 GB, less than 2g required for the cache, and .5 for other server processing requirements.
For the examples from the documentation:
"if you have one universe with 500,000 users and 500,000 roles, set maxElementsInMemory to 3,000,000 elements."
This environment would require roughly a 9gb Xmx Memory allocation.
"If you have two universes, each with 500,000 users and 500,000 roles, set maxElementsInMemory to 6,000,000 elements."
This environment would require roughly an 18gb Xmx memory allocation.
Additional Information
The ehcache-sageDal.xml file mentioned in the documentation can be found in: \CA\RCM\Server\eurekify-jboss\standalone\deployments\eurekify.war\WEB-INF\classes\
And to clarify this statement in the documentation:
"When using a 64bit JDK, and the available memory is greater than 1400M, set the -Xms parameter to use all available memory."
Simply states that the Xms and Xmx values should match in 64bit environments.
Attachments
Feedback
