Error : java.lang.OutOfMemoryError: Java heap space in AdminUI WAMUI

Products

CA Single Sign On Secure Proxy Server (SiteMinder) CA Single Sign On Agents (SiteMinder) CA Single Sign On Federation (SiteMinder) CA Single Sign On SOA Security Manager (SiteMinder) SITEMINDER

Issue/Introduction

When running an AdminUI and when trying to login to the Adminui from
browser, the login fails and the AdminUI logs report error :

  Unable to establish administration context.
  Caught unexpected Throwable: java.lang.OutOfMemoryError: Java heap space

In server.log that the JBoss Server has memory problem :

  2019-12-03 10:24:49,222 [ERROR]
  com.ca.siteminder.framework.xps.security.AdministratorRelationship
  [] - Failed to fetch administrator record for user: [myadmin]
  uid=myadmin,ou=people,dc=XXXX,dc=com

  com.ca.siteminder.uiagent.UIAgentException: Cause:
  java.lang.RuntimeException: Unable to establish administration context.

  2019-12-03 11:08:29,791 ERROR [org.hornetq.core.client] (Thread-1
  (HornetQ-server-HornetQServerImpl::serverUUID=XXXXXX-XXXX-XXXX-XXXX-XXXXXXXX-XXXXXXXX))
  HQ214017: Caught unexpected Throwable: java.lang.OutOfMemoryError: Java heap space

In standalone.conf, the following setting are in place, but the issue
still persists :

  JAVA_OPTS="-Xms1024m -Xmx1536m -XX:MaxMetaspaceSize=256M -Djava.net.preferIPv4Stack=true"

Environment

Environment:

Applicable to all the supported 12.8 and 12.8.x releases.

Cause

It looks the problem is known in the WildFly 8.2.0, as there's a
memory leak.

The AdminUI reports starting fine and shows BufferOverflowException
regularly :

server.log

  2019-12-10 13:46:24,242 INFO [org.jboss.as] (MSC service thread 1-7)
  JBAS015899: WildFly 8.2.0.Final "Tweek" starting

  2019-12-10 13:46:26,061 INFO [org.xnio] (MSC service thread 1-1)
  XNIO version 3.3.0.Final

  2019-12-12 17:28:25,980 WARN [ims.default] (MSC service thread
  1-5) # CA Single Sign-On 12.80.0200.1993

  2019-12-12 17:28:30,640 WARN [ims.default] (MSC service thread
  1-5) * Startup Step 29 : Attempting to start
  ApplicationContextInitializer plug-ins

  2019-12-12 17:28:30,640 WARN [ims.default] (MSC service thread 1-5)
  ---- CA IAM FW Startup Sequence Complete. ----

  2019-12-12 17:29:50,972 [INFO] com.ca.siteminder.uiagent.Connector
  [] - Establishing agent API connection for xxxxxxx

  2019-12-12 18:59:41,949 ERROR [io.undertow.request] (default
  task-39) Undertow request failed HttpServerExchange{ POST
  /iam/siteminder/app/ui7/index.jsp}:
  java.nio.BufferOverflowException

  2019-12-12 19:13:18,123 ERROR [io.undertow.request] (default
  task-32) Undertow request failed HttpServerExchange{ POST
  /iam/siteminder/app/ui7/index.jsp}:
  java.nio.BufferOverflowException

  2019-12-12 19:15:15,699 ERROR [io.undertow.request] (default
  task-17) Undertow request failed HttpServerExchange{ POST
  /iam/siteminder/app/ui7/index.jsp}:
  java.nio.BufferOverflowException

   at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:363) [rt.jar:1.8.0_202]

until the OutOfMemoryError

  2019-12-13 00:46:09,831 ERROR [ims.tmt.persistence] (Thread-274
  (HornetQ-client-global-threads-908099883)) PersistenceProvider:
  JMS:ID:XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX: updateObject: exception
  updating objects: java.sql.BatchUpdateException: Java exception:
  'Java heap space: java.lang.OutOfMemoryError'.

   at org.apache.derby.impl.jdbc.EmbedStatement.executeBatch(Unknown Source)

This is fixed in WildFly 10.0.0 (1).

Some people report having fixed the memory leak by setting this
parameter (2)(3).

Resolution

In AdminUI standalone.conf, add

  -Djboss.remoting.pooled-buffers=false

in line :

  JAVA_OPTS="-Xms1024m -Xmx1536m -XX:MaxMetaspaceSize=256M -Djava.net.preferIPv4Stack=true"

and increase the value of Xmx to have this configuration :

  JAVA_OPTS="-Xms1024m -Xmx4096m -XX:MaxMetaspaceSize=256M -Djava.net.preferIPv4Stack=true -Djboss.remoting.pooled-buffers=false"

Additional Information

(1)

    OutOfMemory DirectByteBuffer XNIO

      The XNIO patch fixes a bug which could cause this error, which will
      be in Wildfly 10.0.0.CR1

    https://issues.redhat.com/browse/WFLY-4696?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&showAll=true

(2)

    Memory Leak in Wildfly 8.2 on remote EJB access from Tomcat 7

     "-Djboss.remoting.pooled-buffers=false"

    https://developer.jboss.org/thread/251919

(3)

    JMX Wildfly 8.2.0 Final
    https://developer.jboss.org/thread/250731