AdminUI Login Fails with "java.lang.OutOfMemoryError: Java heap space" Due to WildFly 8.2.0 Memory Leak

Products

• SITEMINDER
• CA Single Sign-On

Issue/Introduction

When running AdminUI and trying to log in from the browser, the login fails and the AdminUI logs report the following error:

Unable to establish administration context.
Caught unexpected Throwable: java.lang.OutOfMemoryError: Java heap space

The server.log shows that the JBoss server has a memory problem:

2019-12-03 10:24:49,222 [ERROR] com.ca.siteminder.framework.xps.security.AdministratorRelationship [] - Failed to fetch administrator record for user: [<uid>] uid=<uid>,dc=example,dc=com
com.ca.siteminder.uiagent.UIAgentException: Cause: java.lang.RuntimeException: Unable to establish administration context.
2019-12-03 11:08:29,791 ERROR [org.hornetq.core.client] (Thread-1 (HornetQ-server-HornetQServerImpl::serverUUID=<value>)) HQ214017: Caught unexpected Throwable: java.lang.OutOfMemoryError: Java heap space

In standalone.conf, the following settings are in place, but the issue still persists:

JAVA_OPTS="-Xms1024m -Xmx1536m -XX:MaxMetaspaceSize=256M -Djava.net.preferIPv4Stack=true"

Cause

This is a known problem in WildFly 8.2.0, caused by a memory leak.

The AdminUI starts up successfully, but the logs regularly show a BufferOverflowException:

server.log

2019-12-10 13:46:24,242 INFO [org.jboss.as] (MSC service thread 1-7) JBAS015899: WildFly 8.2.0.Final "Tweek" starting
2019-12-10 13:46:26,061 INFO [org.xnio] (MSC service thread 1-1) XNIO version 3.3.0.Final
2019-12-12 17:28:25,980 WARN [ims.default] (MSC service thread 1-5) # CA Single Sign-On 12.80.0200.1993
2019-12-12 17:28:30,640 WARN [ims.default] (MSC service thread 1-5) * Startup Step 29 : Attempting to start ApplicationContextInitializer plug-ins
2019-12-12 17:28:30,640 WARN [ims.default] (MSC service thread 1-5) ---- CA IAM FW Startup Sequence Complete. ----
2019-12-12 17:29:50,972 [INFO] com.ca.siteminder.uiagent.Connector [] - Establishing agent API connection for <value>
2019-12-12 18:59:41,949 ERROR [io.undertow.request] (default task-39) Undertow request failed HttpServerExchange{ POST /iam/siteminder/app/ui7/index.jsp}: java.nio.BufferOverflowException
2019-12-12 19:13:18,123 ERROR [io.undertow.request] (default task-32) Undertow request failed HttpServerExchange{ POST /iam/siteminder/app/ui7/index.jsp}: java.nio.BufferOverflowException
2019-12-12 19:15:15,699 ERROR [io.undertow.request] (default task-17) Undertow request failed HttpServerExchange{ POST /iam/siteminder/app/ui7/index.jsp}: java.nio.BufferOverflowException
   at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:363) [rt.jar:1.8.0_202] until the OutOfMemoryError
2019-12-13 00:46:09,831 ERROR [ims.tmt.persistence] (Thread-274 (HornetQ-client-global-threads-908099883)) PersistenceProvider: JMS:ID:<value>: updateObject: exception updating objects: java.sql.BatchUpdateException: Java exception: 'Java heap space: java.lang.OutOfMemoryError'.
   at org.apache.derby.impl.jdbc.EmbedStatement.executeBatch(Unknown Source)

This issue is fixed in WildFly 10.0.0 (1) and later.

Resolution

Upgrade the SiteMinder environment and AdminUI to version 12.9 to obtain WildFly version 24.x, which resolves this issue.

Check the AdminUI's currently running WildFly version using the following command (2):

For Windows:

c:> .\standalone.bat -V

For Linux:

# ./standalone.sh -V

Additional Information

1. OutOfMemory DirectByteBuffer XNIO
2. How to verify the WildFly version shipped with AdminUI