Identity Manager error "java.lang.OutOfMemoryError: GC overhead limit exceeded" found in server.log

Products

• CA Identity Manager
• CA Identity Governance
• CA Identity Portal
• CA Risk Analytics
• CA Secure Cloud SaaS - Arcot A-OK (WebFort)
• CLOUDMINDER ADVANCED AUTHENTICATION
• CA Secure Cloud SaaS - Advanced Authentication
• CA Secure Cloud SaaS - Identity Management
• CA Secure Cloud SaaS - Single Sign On

Issue/Introduction

This error can occur in a clustered environments where memory is exceeded.  When this occurs the nodes may stop taking traffic.  You may find a restart of services resolves the issue for a time, but a performance setting will need to be adjusted for a permanent resolution.

2017-04-23 22:39:32,850 ERROR [org.jgroups.util.TimeScheduler3] (Timer runner-1,shared=tcp) failed submitting task to thread pool: java.lang.OutOfMemoryError: GC overhead limit exceeded 

2017-04-23 22:40:29,256 ERROR [org.jgroups.util.TimeScheduler3] (Timer runner-1,shared=udp) failed submitting task to thread pool: java.lang.OutOfMemoryError: GC overhead limit exceeded 

2017-04-23 22:40:25,788 ERROR [org.hornetq.core.client] (Thread-20 (HornetQ-server-HornetQServerImpl::serverUUID=<IDM_UUID_STRING>)) HQ214017: Caught unexpected Throwable: java.lang.OutOfMemoryError: GC overhead limit exceeded 

2017-04-23 22:40:44,944 ERROR [org.xnio.listener] (default I/O-3) XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError: GC overhead limit exceeded 

How to resolve "GC overhead limit exceeded" error

Environment

Identity Manager

Cause

The out of memory error indicates that the Application Server ran out of available memory to perform actions requested by the deployed application, in this case Identity Manager. 

Resolution

The solution is to increase memory allocation.  

Our product documentation should be referenced to make these adjustments:
Performance Tuning

For example if running Identity Manager on Jboss and the system requires more than 4gb of memory the JAVA_OPTS setting should be set similar to:

JAVA_OPTS="-Xms8g -Xmx8g -XX:+UseG1GC -XX:ConcGCThreads=12 -XX:ParallelGCThreads=22 -XX:MaxGCPauseMillis=1000 -XX:InitiatingHeapOccupancyPercent=40 -XX:G1HeapWastePercent=2 -XX:G1ReservePercent=15 -XX:+UnlockExperimentalVMOptions -XX:G1OldCSetRegionThresholdPercent=15 -XX:G1MixedGCLiveThresholdPercent=90 -XX:G1NewSizePercent=20 -XX:G1MaxNewSizePercent=25 -Djava.net.preferIPv4Stack=true" JAVA_OPTS="$JAVA_OPTS -Djboss.modules.system.pkgs=$JBOSS_MODULES_SYSTEM_PKGS -Djava.awt.headless=true" JAVA_OPTS="$JAVA_OPTS -Djboss.modules.policy-permissions=true"