STUCK_THREADS related error in WebLogic console logs for Risk Authentication
Article ID: 242407
Updated On:
Products
Issue/Introduction
After running AA transactions for a while, timeout errors are encountered when invoking Advanced Authentication APIs. The issue is remediated via restart of Weblogic application server. After running for a while the issue resurfaces again.
In the Webloic Console logs one can see the error below.
Symtom{STUCK_THREADS,HIGH,ThreadPoolRuntime,ThreadPool has stuck threads}
Error log from stdout.log
"[STUCK] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" #29 daemon prio=1 os_prio=0 tid=0x00007f0945865000 nid=0x3f0e in Object.wait() [0x00007f0971eaf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1118)
- locked <0x00000000a4b71c98> (a org.apache.commons.pool.impl.GenericObjectPool$Latch)
at com.ca.advauth.util.ServiceConnection.getAdminStubSvcInstance(ServiceConnection.java:424)
at com.ca.advauth.risk.impl.LocationAndConnectionInfoImpl.fetchLocationAndConnection(LocationAndConnectionInfoImpl.java:80)
at com.ca.advauth.risk.services.LocationAndConnectionService.fetchLocationAndConnectionInfo(LocationAndConnectionService.java:87)
at sun.reflect.GeneratedMethodAccessor264.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:295)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:260)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:137)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:353)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:250)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3793)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3763)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:344)
at weblogic.security.service.SecurityManager.runAsForUserCode(SecurityManager.java:197)
at weblogic.servlet.provider.WlsSecurityProvider.runAsForUserCode(WlsSecurityProvider.java:203)
at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:71)
at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2451)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2299)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2277)
at weblogic.servlet.internal.ServletRequestImpl.runInternal(ServletRequestImpl.java:1710)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1670)
at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:272)
at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352)
at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337)
at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57)
at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:655)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:420)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:360)
"weblogic.timers.TimerThread" #27 daemon prio=9 os_prio=0 tid=0x00007f0938930800 nid=0x3f0c in Object.wait() [0x00007f09354f9000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at weblogic.timers.internal.TimerThread.doWait(TimerThread.java:115)
- locked <0x00000000a042cbf0> (a java.lang.Object)
at weblogic.timers.internal.TimerThread.access$000(TimerThread.java:21)
at weblogic.timers.internal.TimerThread$Thread.run(TimerThread.java:291)
"weblogic.time.TimeEventGenerator" #26 daemon prio=9 os_prio=0 tid=0x00007f093892f800 nid=0x3f0b in Object.wait() [0x00007f09355fa000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:286)
- locked <0x00000000a04179e0> (a weblogic.time.common.internal.TimeTable)
at weblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java:121)
at java.lang.Thread.run(Thread.java:748)
Environment
Release : All supported releases of Advanced Authentication
Component : Risk Authentication
Cause
CA Strong Authentication server's DB connections can get exhausted if not properly configured.
Resolution
Please restart the AA services and Applications after instating the two changes below:
1. As shown below, please configure CA Strong Authentication server's database connections by updating the arcotcommon.ini file in <ARCOT_HOME>/conf directory
MinConnections=16
MaxConnections=64
IncConnections=2
MaxIdleConnections=16
2. Run this SQL Database query such that any lock held on internal device table or user table is released in a timely fashion for optimal CA risk Authentication transaction processing.
update ARRFCONFIGURATION set value=5 where category='GLOBAL' and name='DEVICELOCKEDTIMEDIFF';
update ARRFCONFIGURATION set value=5 where category='GLOBAL' and name='USERLOCKEDTIMEDIFF';
3. Please also ensure tables that grow quickly such as ARRFSYSAUDITLOG table (that are recommended to be trimmed) are not allowed to grow too large.
Additional Information
None.
Feedback
