STUCK_THREADS related error in WebLogic console logs for Risk Authentication

book

Article ID: 242407

calendar_today

Updated On:

Products

CA Risk Authentication CA Advanced Authentication

Issue/Introduction

After running AA transactions for a while, timeout errors are encountered when invoking Advanced Authentication APIs. The issue is remediated via restart of Weblogic application server. After running for a while the issue resurfaces again.

In the Webloic Console logs one can see the error below.

                   Symtom{STUCK_THREADS,HIGH,ThreadPoolRuntime,ThreadPool has stuck threads}

Error log from stdout.log 

"[STUCK] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" #29 daemon prio=1 os_prio=0 tid=0x00007f0945865000 nid=0x3f0e in Object.wait() [0x00007f0971eaf000]

   java.lang.Thread.State: WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        at java.lang.Object.wait(Object.java:502)

        at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1118)

        - locked <0x00000000a4b71c98> (a org.apache.commons.pool.impl.GenericObjectPool$Latch)

        at com.ca.advauth.util.ServiceConnection.getAdminStubSvcInstance(ServiceConnection.java:424)

        at com.ca.advauth.risk.impl.LocationAndConnectionInfoImpl.fetchLocationAndConnection(LocationAndConnectionInfoImpl.java:80)

        at com.ca.advauth.risk.services.LocationAndConnectionService.fetchLocationAndConnectionInfo(LocationAndConnectionService.java:87)

        at sun.reflect.GeneratedMethodAccessor264.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)

        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)

        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)

        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205)

        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)

        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)

        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)

        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)

        at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)

        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)

        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)

        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)

        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)

        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)

        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)

        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)

        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)

        at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)

        at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)

        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)

        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)

        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)

        at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:295)

        at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:260)

        at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:137)

        at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:353)

        at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:250)

        at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3793)

        at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3763)

        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:344)

        at weblogic.security.service.SecurityManager.runAsForUserCode(SecurityManager.java:197)

        at weblogic.servlet.provider.WlsSecurityProvider.runAsForUserCode(WlsSecurityProvider.java:203)

        at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:71)

        at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2451)

        at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2299)

        at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2277)

        at weblogic.servlet.internal.ServletRequestImpl.runInternal(ServletRequestImpl.java:1710)

        at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1670)

        at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:272)

        at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352)

        at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337)

        at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57)

        at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)

        at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:655)

        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:420)

        at weblogic.work.ExecuteThread.run(ExecuteThread.java:360)

 

"weblogic.timers.TimerThread" #27 daemon prio=9 os_prio=0 tid=0x00007f0938930800 nid=0x3f0c in Object.wait() [0x00007f09354f9000]

   java.lang.Thread.State: TIMED_WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        at weblogic.timers.internal.TimerThread.doWait(TimerThread.java:115)

        - locked <0x00000000a042cbf0> (a java.lang.Object)

        at weblogic.timers.internal.TimerThread.access$000(TimerThread.java:21)

        at weblogic.timers.internal.TimerThread$Thread.run(TimerThread.java:291)

 

"weblogic.time.TimeEventGenerator" #26 daemon prio=9 os_prio=0 tid=0x00007f093892f800 nid=0x3f0b in Object.wait() [0x00007f09355fa000]

   java.lang.Thread.State: TIMED_WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:286)

        - locked <0x00000000a04179e0> (a weblogic.time.common.internal.TimeTable)

        at weblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java:121)

        at java.lang.Thread.run(Thread.java:748)

 

 

Cause

CA Strong Authentication server's DB connections can get  exhausted if not properly configured.

Environment

Release : All supported releases of Advanced Authentication

Component : Risk Authentication

Resolution

Please restart  the AA services and Applications after instating the two changes below:

1. As shown below, please configure CA Strong Authentication server's  database connections by updating the arcotcommon.ini file in <ARCOT_HOME>/conf directory 

MinConnections=16
MaxConnections=64
IncConnections=2
MaxIdleConnections=16

2. Run this SQL Database query such that any lock held on internal device table or user table is released in a timely fashion for optimal CA risk Authentication transaction processing.

update ARRFCONFIGURATION set value=5 where category='GLOBAL' and name='DEVICELOCKEDTIMEDIFF';
update ARRFCONFIGURATION set value=5 where category='GLOBAL' and name='USERLOCKEDTIMEDIFF';

3. Please also ensure tables that grow quickly such as ARRFSYSAUDITLOG table (that are recommended to be trimmed) are not allowed to grow too large.

Additional Information

None.