Symantec Identity Manager - Application consumes more memory than -Xmx setting, CPU climbs to 100%, application becomes unresponsive requiring a restart.

Products

• CA Identity Manager

Issue/Introduction

This issue is describing a rare memory leak in 14.3 and 14.4 when running on the Jboss 7.2 or Wildfly 15 application servers.   

The issue presents as a continuous climb of memory up to and beyond the -Xmx max memory allocation in the standalone.conf.xml file.   Once the max memory is exceeded, CPU usage will climb, eventually reaching 100% usage, at which point the product begins to slow overall, tasks will linger in-progress, and eventually, IDM will hang and become unavailable to end users.   


A restart will clear the issue temporarily, but the pattern continues, memory being utilized but not released until the max memory allocation is exceeded. 

 

Environment

Release : 14.3, 14.4 after updating to JBOSS 7.2 or Wildfly 15. 

Cause

This is due to a memory leak in the netty jar file.  
We have not identified the specific use case, or actions that lead to this, and only a small number of clients have faced this behavior. 

Resolution

Note: This is an application layer problem, if application support is available then Broadcom recommends discussing this issue with them as they can assist with the help dump analysis. 

To try to identify if an IDM deployment is experiencing this issue you can use Heap Dumps. 

When the system is at a high memory and cpu uase, take a java heap dump.
There are multiple ways to collect the heap dump, the simplest would be to use the use the jmap tool which is provided by the JDK/bin folder.

jmap -dump:format=b,file=<file-path> <pid>

 
The resulting heap dump can be reviewed in a profiler tool like JVisualVM or VisualVM.   Check for the memory usage of the netty object, netty-all-4.1.25.Final.jar.  If we see netty-all-4.1.25.Final.jar consuming large amounts of memory then the attached fix should help.

The new netty-all-4.1.68.Final.jar can be downloaded from here:

Netty/All In One » 4.1.68.Final

netty-all JAR 4.1.68.Final with all dependencies

 

1.    Stop the Identity Manager Server.

2.    Take a backup and remove the existing netty-all-4.1.25.Final.jar and module.xml files from <APPSERVER_HOME>/modules/system/layers/base/io/netty/main/ folder

3.    Copy the new netty-all-4.1.68.Final.jar into <APPSERVER_HOME>/modules/system/layers/base/io/netty/main/ folder.

4.    Update the module.xml file located in <APPSERVER_HOME>/modules/system/layers/base/io/netty/main/ as follows (an example module.xml is attached to this defect) 

<?xml version="1.0" encoding="UTF-8"?>

<!--

  ~ JBoss, Home of Professional Open Source.

  ~ Copyright 2010, Red Hat, Inc., and individual contributors

  ~ as indicated by the @author tags. See the copyright.txt file in the

  ~ distribution for a full listing of individual contributors.

  ~

  ~ This is free software; you can redistribute it and/or modify it

  ~ under the terms of the GNU Lesser General Public License as

  ~ published by the Free Software Foundation; either version 2.1 of

  ~ the License, or (at your option) any later version.

  ~

  ~ This software is distributed in the hope that it will be useful,

  ~ but WITHOUT ANY WARRANTY; without even the implied warranty of

  ~ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU

  ~ Lesser General Public License for more details.

  ~

  ~ You should have received a copy of the GNU Lesser General Public

  ~ License along with this software; if not, write to the Free

  ~ Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA

  ~ 02110-1301 USA, or see the FSF site: http://www.fsf.org.

  -->

<module name="io.netty" xmlns="urn:jboss:module:1.5">

    <properties>

        <property name="jboss.api" value="private"/>

    </properties>

 

    <resources>

        <resource-root path="netty-all-4.1.68.Final.jar"/>

    </resources>

 

    <dependencies>

        <module name="javax.api"/>

        <module name="sun.jdk"/>

<module name="org.javassist" optional="true"/>

<module name="com.ca.iam.log4j2.api"/>

    </dependencies>

</module>



5.    Restart Identity Manager server.

 

 

Additional Information

All clients can perform this update.  Updating the netty jar to netty-all-4.1.68.Final.jar is very low risk and should not cause any problems within Identity Manager. 



HF_UNIVERSAL-20230628121233-V2_NETTY_UPGRADE.tgz is the VAPP 14.4 fix and is attached below.