Error: java.lang.OutOfMemoryError: Java heap space in AdminUI WAMUI

Products

SITEMINDER CA Single Sign-On

Issue/Introduction

When running an AdminUI and when trying to log in to the Adminui from the browser, the login fails and the AdminUI logs report the error:

Unable to establish administration context.
Caught unexpected Throwable: java.lang.OutOfMemoryError: Java heap space

The server.log reports that the JBoss Server has a memory problem:

2019-12-03 10:24:49,222 [ERROR] com.ca.siteminder.framework.xps.security.AdministratorRelationship [] - Failed to fetch administrator record for user: [<uid>] uid=<uid>,dc=example,dc=com

com.ca.siteminder.uiagent.UIAgentException: Cause: java.lang.RuntimeException: Unable to establish administration context.

2019-12-03 11:08:29,791 ERROR [org.hornetq.core.client] (Thread-1 (HornetQ-server-HornetQServerImpl::serverUUID=<value>)) HQ214017: Caught unexpected Throwable: java.lang.OutOfMemoryError: Java heap space

In standalone.conf, the following settings are in place, but the issue still persists:

JAVA_OPTS="-Xms1024m -Xmx1536m -XX:MaxMetaspaceSize=256M -Djava.net.preferIPv4Stack=true"

Cause

The problem is known in the WildFly 8.2.0, as there is a memory leak.

The AdminUI reports start fine and show BufferOverflowException regularly:

server.log

2019-12-10 13:46:24,242 INFO [org.jboss.as] (MSC service thread 1-7) JBAS015899: WildFly 8.2.0.Final "Tweek" starting
2019-12-10 13:46:26,061 INFO [org.xnio] (MSC service thread 1-1) XNIO version 3.3.0.Final
2019-12-12 17:28:25,980 WARN [ims.default] (MSC service thread 1-5) # CA Single Sign-On 12.80.0200.1993
2019-12-12 17:28:30,640 WARN [ims.default] (MSC service thread 1-5) * Startup Step 29 : Attempting to start ApplicationContextInitializer plug-ins
2019-12-12 17:28:30,640 WARN [ims.default] (MSC service thread 1-5) ---- CA IAM FW Startup Sequence Complete. ----
2019-12-12 17:29:50,972 [INFO] com.ca.siteminder.uiagent.Connector [] - Establishing agent API connection for <value>
2019-12-12 18:59:41,949 ERROR [io.undertow.request] (default task-39) Undertow request failed HttpServerExchange{ POST /iam/siteminder/app/ui7/index.jsp}: java.nio.BufferOverflowException
2019-12-12 19:13:18,123 ERROR [io.undertow.request] (default task-32) Undertow request failed HttpServerExchange{ POST /iam/siteminder/app/ui7/index.jsp}: java.nio.BufferOverflowException
2019-12-12 19:15:15,699 ERROR [io.undertow.request] (default task-17) Undertow request failed HttpServerExchange{ POST /iam/siteminder/app/ui7/index.jsp}: java.nio.BufferOverflowException
   at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:363) [rt.jar:1.8.0_202] until the OutOfMemoryError 
2019-12-13 00:46:09,831 ERROR [ims.tmt.persistence] (Thread-274 (HornetQ-client-global-threads-908099883)) PersistenceProvider: JMS:ID:<value>: updateObject: exception updating objects: java.sql.BatchUpdateException: Java exception: 'Java heap space: java.lang.OutOfMemoryError'.
   at org.apache.derby.impl.jdbc.EmbedStatement.executeBatch(Unknown Source)

This is fixed in WildFly 10.0.0 (1).

Some people report having fixed the memory leak by setting this parameter (2)(3).

Resolution

In AdminUI standalone.conf, add

-Djboss.remoting.pooled-buffers=false

in line:

JAVA_OPTS="-Xms1024m -Xmx1536m -XX:MaxMetaspaceSize=256M -Djava.net.preferIPv4Stack=true"

and increase the value of Xmx to have this configuration:

JAVA_OPTS="-Xms1024m -Xmx4096m -XX:MaxMetaspaceSize=256M -Djava.net.preferIPv4Stack=true -Djboss.remoting.pooled-buffers=false"