AWI Tomcat or Jetty goes down with java heap space error
search cancel

AWI Tomcat or Jetty goes down with java heap space error

book

Article ID: 243065

calendar_today

Updated On:

Products

CA Automic Workload Automation - Automation Engine CA Automic One Automation

Issue/Introduction

Automic web interface performance degrades at times. The following exception is captured in the tomcat's std error file (%CATALINA_BASE%\logs\apachetomcat8-stderr.<date>.log)
28-May-2022 10:38:50.639 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run Unexpected death of background thread [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
 java.lang.OutOfMemoryError: Java heap space
28-May-2022 14:06:18.698 INFO [Thread-51] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-nio-8080"]uler-1554" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "UC4 System=AUTOMIC, Client=0100, User=AUTOMIC, SessionID=0000000004882026" java.lang.OutOfMemoryError: Java heap space
Exception in thread "AsyncFileHandlerWriter-664740647" java.lang.OutOfMemoryError: Java heap space
Exception in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" java.lang.OutOfMemoryError: Java heap space

Restarting the Tomcat seems to work around the problem. However, it returns in a few hours of continuous operation.

With v21 and superior Jetty AWI, the issue can also manifest the same way.

The following kind of errors may appear in the AWI Log <Jetty_AWI_Path>\osgi-tmp\0\HOSTNAME_ECC_LOG.00.txt: 

2023-03-24 16:39:39,906 qtp265294114-43642     [ERROR] NOLOGIN/- NOUI   [com.uc4.ecc.framework.entrypoint.vaadin14.V14VaadinServlet] - ERROR IN SERVET
java.lang.OutOfMemoryError: Java heap space
2023-03-24 16:45:36,724 qtp265294114-43991     [WARN ] NOLOGIN/- NOUI   [com.uc4.ecc.framework.core.blacklist.VersionsBlacklist] - Error while evaluating version from list of osgi bundles.
java.lang.OutOfMemoryError: Java heap space

Environment

Release : 12.3 and 21.X

Component : Automation Engine

Sub-Component: Automic Web Interface (AWI) 

Issue may occur on either Tomcat or on Jetty AWI

Cause

The problem is due to a wrong memory configuration of the Web Server hosting AWI (Tomcat or Jetty) not having enough heap size to process the user requests.

Resolution

  • If AWI is hosted on a Tomcat Server:

The heap size for the tomcat needs to be increased to resolve the problem. By default, Tomcat allocates a low amount of memory to an application (256MB by default). This is not sufficient for AWI to process the concurrent user requests.

Windows:

      1. Go to the bin folder in your Tomcat installation (...Apache Software Foundation\Tomcat<version>\bin).
      2. Right-click the Commons Daemon Service Manager file (tomcat9w.exe) file and, Run as administrator.
      3. On the Java tab, set the heap size by setting the Initial and Maximum memory pool to the maximum possible on your system, for example, 8192MB (-Xmx8192m).

Alternatively on Windows, below is an example (setenv.bat) in Windows to increase the PermGen Space and the Heapsize. Automic recommends at least 256 m for PermGen Space and as much as possible for Heap size .   

set CATALINA_OPTS= -Xmx8192m -XX:MaxPermSize=256m set JAVA_OPTS= -XX:MaxPermSize=256m set JAVA_HOME="C:\Automic\External.Resources\JDK\jdk1.8.0_45" set JRE_HOME="C:\Automic\External.Resources\JDK\jdk1.8.0_45\jre"
Your command file (catalina.bat/Catalina.sh) already has a statement as below:   

if exist "%CATALINA_HOME%\bin\setenv.bat" call "%CATALINA_HOME%\bin\setenv.bat"  

Linux:

In the CATALINA_OPTS environment variable, set the heap space to the maximum possible on your system, for example, 8192MB.

export CATALINA_OPTS=-Xms8192m -Xmx8192m

On Linux environments, the variable can be set in the tomcat server startup script ($CATALINA_BASE/bin/start.sh).

Restart the tomcat service and verify the heap message in the Catalina log file ($CATALINA_BASE/log/catalina.log)  reflects the updated value.

02-Jun-2022 22:42:46.298 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms128m
02-Jun-2022 22:42:46.298 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx8192m
02-Jun-2022 22:42:46.303 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded Apache Tomcat Native library [1.2.32] using APR version [1.7.0].
02-Jun-2022 22:42:46.303 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true], UDS [true].
02-Jun-2022 22:42:46.303 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
02-Jun-2022 22:42:46.309 INFO [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL successfully initialized [OpenSSL 1.1.1n  15 Mar 2022]
02-Jun-2022 22:42:46.824 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]

  • If using AWI with the Bundled Jetty Application Server:

Check the command line used to start the Jetty AWI in the Service Manager Dialog.
If the setting -Xmx is not set Java will allocate as much memory as possible at System Level to the JVM:

If this is not the case and a setting like -Xmx2G has been used and the error continues to appear, please increase the setting to a bigger value (provided that the Operating System has enough Memory), like:

java -Xrs -Xms4G -Xmx4G -jar C:\temp\awi\aa-webui-launcher.jar

Make sure that both init (Xms) and max (Xmx) values are set equally.
Then restart the Jetty AWI via the service manager and check in the log that the parameter has been taken into account:

2023-03-22 13:57:30,007 olve Thread (Bundle 3) [INFO ] NOLOGIN/- NOUI   [com.uc4.webui.logback.osgi.EnvironmentLogger] - JVM Argument  -Xmx4G                                            

Additional Information

There is no recommendation on how much memory would it require to process the user request. It totally depends on the user-specific operations and concurrent requests.

Generally, it is a good idea to start with 8GB of the max heap and further increase it if required.

The documentation has some information about setting max memory as well: https://docs.automic.com/documentation/WEBHELP/English/all/components/DOCU/21.0/Automic%20Automation%20Guides/Content/Installation_Manual/AWI/awi_install_tomcat_donotdelete.htm#IncreaseMemoryTomcat

Powered by Wolken Software