Repeated CleanupStaleConnections failed Exception Errors and high memory consumption

Products

• CA Automic Workload Automation - Automation Engine
• CA Automic One Automation

Issue/Introduction

We have repeated occurrences of " cleanupStaleConnections failed" exception errors below in AWI logs.

Looks like stale connections are not being cleaned up in both Apache Tomcat and the Jetty launcher, this is the message that appears in AWI log:

[ERROR] NOLOGIN/- NOUI   [com.uc4.ecc.framework.core.inject.BaseScopeManager] - cleanupStaleConnections failed
java.lang.reflect.UndeclaredThrowableException: null
 at com.sun.proxy.$Proxy36.run(Unknown Source)
 at com.uc4.ecc.framework.core.inject.EntityCache.cleanup(EntityCache.java:65)
 at com.uc4.ecc.framework.core.inject.BaseScopeManager.destroyScopeData(BaseScopeManager.java:116)
 at com.uc4.ecc.framework.core.inject.ContextAwareScope.destroyScopeData(ContextAwareScope.java:58)
 at com.uc4.ecc.framework.core.inject.ContextAwareScope.destroyScopeData(ContextAwareScope.java:19)
 at com.uc4.ecc.framework.core.inject.BaseScopeManager.cleanup(BaseScopeManager.java:41)
 at com.uc4.ecc.framework.entrypoint.shutdown.SessionCleanupService.lambda$cleanupScopes$0(SessionCleanupService.java:32)
 at java.base/java.util.Arrays$ArrayList.forEach(Arrays.java:4390)
 at com.uc4.ecc.framework.entrypoint.shutdown.SessionCleanupService.cleanupScopes(SessionCleanupService.java:32)
 at com.uc4.ecc.framework.entrypoint.shutdown.SessionCleanupService.access$000(SessionCleanupService.java:14)
 at com.uc4.ecc.framework.entrypoint.shutdown.SessionCleanupService$1.run(SessionCleanupService.java:43)
 at java.base/java.util.TimerThread.mainLoop(Timer.java:556)
 at java.base/java.util.TimerThread.run(Timer.java:506)
Caused by: java.lang.reflect.InvocationTargetException: null
 at java.base/jdk.internal.reflect.GeneratedMethodAccessor121.invoke(Unknown Source)
 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.base/java.lang.reflect.Method.invoke(Method.java:566)
 at com.uc4.webui.common.observable.InternalObservable.lambda$new$0(InternalObservable.java:26)
 ... 13 common frames omitted
Caused by: java.lang.IllegalStateException: May only be called from a context-aware thread
 at com.google.common.base.Preconditions.checkState(Preconditions.java:502)
 at com.uc4.ecc.framework.core.async.push.PushRequestCoordinator.checkCorrectThreadForGuiOperations(PushRequestCoordinator.java:39)
 at com.uc4.ecc.framework.core.async.BaseRequestCoordinator.cancelInGuiThread(BaseRequestCoordinator.java:319)
 at com.uc4.webui.common.async.IRequestCoordinator.cancelInGuiThread(IRequestCoordinator.java:50)
 at com.uc4.ecc.plugins.ae.perspectives.processmonitoring.content.monitor.GraphicalWorkflowMonitorController.onClose(GraphicalWorkflowMonitorController.java:131)
 at com.uc4.ecc.plugins.ae.perspectives.processmonitoring.content.monitor.MonitorContentViewController.cleanup(MonitorContentViewController.java:1953)
 at com.uc4.ecc.framework.entrypoint.ui.ContentService.unregister(ContentService.java:102)
 at com.uc4.ecc.framework.entrypoint.ui.ContentService.lambda$registerContent$2(ContentService.java:90)
 ... 17 common frames omitted

This can cause outages in some cases where, depending on user behavior (not logging out before closing browser tabs, etc...), the memory dedicated to the AWI could be fully used and not released, so there is high memory consumption and CPU usage.

Environment

Release : 21.x

Component: CA Automic Workload Automation

Sub-Component: Automic.Web.Interface (AWI)

Cause

The issue has been confirmed to be a bug by our Dev team.

Resolution

Workaround:

Restart the AWI service (Tomcat or Jetty) as memory is filling up.

Solution:

Update to a fix version listed below or a newer version if available.

Fix version:
Component(s): Automic Web Interface (AWI)
Automation.Engine and AWI 21.0.8 - Available

( Automation.Engine 21.0.7HF1 - Available and AWI 21.0.7 HF3 - Available )

Please note: the AWI and Automation Engine components should be on the same service pack. [Major.Minor.ServicePack]

ATTENTION:

Please make sure that the -Xms startup setting of the AWI is set to a low value such as -Xms1024m , it should NEVER be set to the same value of the setting -Xmx setting (usually advised to be set at -Xmx2048m or -Xms4096m). If the -Xms setting is set to the same value as -Xmx then the outcome may be that the Java process (either Tomcat or Jetty AWI) will never deallocate the memory and AWI performances will be impacted.

Additional Information

Bug ID: AE-33133 (this replaces the old AE-31823 as the correction was not working as expected).