Websphere Application Sessions invalidated when JavaAgent tries to obtain info from the slow/stalled request.

Products

• CA Application Performance Management (APM / Wily / Introscope)

Issue/Introduction

Our Websphere application includes a function for files upload.

This works fine with most of users except for some users (who run over a relatively slower WAN connectivity), the upload fails due to a security violation that causes the session to be invalidated.

Stack trace:
3:39:26.351 0xfa1400    j9trc_aux.0    - jstacktrace:
23:39:26.351 0xfa1400    j9trc_aux.1    - [1] com.ibm.ws.webcontainer.srt.SRTServletRequestThreadData.<init> (SRTServletRequestThreadData.java:62)
23:39:26.351 0xfa1400    j9trc_aux.1    - [2] com.ibm.ws.webcontainer.srt.SRTServletRequestThreadData.getInstance (SRTServletRequestThreadData.java:54) (Compiled Code)
23:39:26.351 0xfa1400    j9trc_aux.1    - [3] com.ibm.ws.webcontainer.srt.SRTServletRequest.getDispatchContext (SRTServletRequest.java:4013) (Compiled Code)
23:39:26.351 0xfa1400    j9trc_aux.1    - [4] com.ibm.ws.webcontainer.srt.SRTServletRequest.getContextPath (SRTServletRequest.java:2755)
23:39:26.351 0xfa1400    j9trc_aux.1    - [5] com.wily.introscope.agent.servlethelper.ServletRequestHelper.getContextPath (ServletRequestHelper.java:143)
23:39:26.351 0xfa1400    j9trc_aux.1    - [6] com.wily.introscope.agent.servlethelper.ServletRequestHelper.get (ServletRequestHelper.java:508)
23:39:26.351 0xfa1400    j9trc_aux.1    - [7] com.wily.introscope.agent.trace.hc2.HttpServletLazyMapProvider.get (HttpServletLazyMapProvider.java:71)
23:39:26.351 0xfa1400    j9trc_aux.1    - [8] com.wily.util.adt.LazyMap.get (LazyMap.java:182)
23:39:26.351 0xfa1400    j9trc_aux.1    - [9] com.wily.introscope.agent.trace.hc2.HttpServletLazyMap.putAllParameters (HttpServletLazyMap.java:191)
23:39:26.351 0xfa1400    j9trc_aux.1    - [10] com.wily.introscope.agent.trace.hc2.HttpServletTracer.IInvocationDataParameterCallback_addParameters (HttpServletTracer.java:875)
23:39:26.351 0xfa1400    j9trc_aux.1    - [11] com.wily.introscope.agent.trace.InvocationData.addParameterCallbacks (InvocationData.java:3557)
23:39:26.351 0xfa1400    j9trc_aux.1    - [12] com.wily.introscope.agent.trace.intelligent.HighPerformanceIntelligentStackHelper.fillComponentParams (HighPerformanceIntelligentStackHelper.java:476)
23:39:26.351 0xfa1400    j9trc_aux.1    - [13] com.wily.introscope.agent.trace.intelligent.HighPerformanceIntelligentStackHelper.getTransanctionSnapShotFromInvocationData (HighPerformanceIntelligentStackHelper.java:599)
23:39:26.351 0xfa1400    j9trc_aux.1    - [14] com.wily.introscope.agent.trace.intelligent.HighPerformanceIntelligentStackHelper.getTransactionSnapshotFromStack (HighPerformanceIntelligentStackHelper.java:320)
23:39:26.351 0xfa1400    j9trc_aux.1    - [15] com.wily.introscope.agent.feature.StallFeatureBase.getMergedTCD (StallFeatureBase.java:122)
23:39:26.351 0xfa1400    j9trc_aux.1    - [16] com.wily.introscope.agent.feature.StallFeatureBase.getTCD (StallFeatureBase.java:93)
23:39:26.351 0xfa1400    j9trc_aux.1    - [17] com.wily.introscope.agent.feature.StallFeatureBase.reportStall (StallFeatureBase.java:70)
23:39:26.351 0xfa1400    j9trc_aux.1    - [18] com.wily.introscope.agent.feature.StallCheckBehaviorCorrect.ITimestampedRunnable_execute (StallCheckBehaviorCorrect.java:66)
23:39:26.351 0xfa1400    j9trc_aux.1    - [19] com.wily.util.heartbeat.IntervalHeartbeat$BehaviorNode.execute (IntervalHeartbeat.java:972) (Compiled Code)
23:39:26.351 0xfa1400    j9trc_aux.1    - [20] com.wily.util.heartbeat.IntervalHeartbeat.executeNextBehaviorAndCalculateSleepTime (IntervalHeartbeat.java:491) (Compiled Code)
23:39:26.351 0xfa1400    j9trc_aux.1    - [21] com.wily.util.heartbeat.IntervalHeartbeat.access$2 (IntervalHeartbeat.java:445) (Compiled Code)
23:39:26.351 0xfa1400    j9trc_aux.1    - [22] com.wily.util.heartbeat.IntervalHeartbeat$HeartbeatRunnable.run (IntervalHeartbeat.java:668)
23:39:26.351 0xfa1400    j9trc_aux.1    - [23] java.lang.Thread.run (Thread.java:812)

..

 

Environment

WebSphere Application Server Liberty profile version 18.0.0.3

Agent 10.7.x

Cause

When the below property is set in Websphere Liberty, Application Sessions terminates when the JavaAgent tries to obtain more information on a “stall”.

 <httpSession cookieSecure="true"
 invalidateOnUnauthorizedSessionRequestException="true" />
<webAppSecurity logoutOnHttpSessionExpire="true" ssoRequiresSSL="true"/>

The JavaAgent heartbeat thread is a timer based thread to process various wily routine jobs, including reporting Stalls. i.e. whenever there is a Stall (slow transaction in this case) detected, the heartbeat thread will try to find out the details about the detected Stall (i.e. HttpServletRequest.getContextPath() to get the context path for the request.) and report it, this is when the issue occurs.

Resolution

Below UserStory has been opened to enhance the JavaAgent code to support this Webphere Liberty property:

US637065: How to deal with IBM JVM property invalidateOnUnauthorizedSessionRequestException=true

 

Current Workarounds:

Option 1)

On IBM side: invalidateOnUnauthorizedSessionRequestException=false

 

Option 2)

On APM side: 
a) Disable Stall in the Agent profile
com.wily.introscope.agent.blame.stall.trace.enabled=false
introscope.agent.deep.stallsnapshot.enabled=false

b)Disable HTTP SErver Tracing
This is done in the toogles-<appserver>.pbd.  
You need to comment (add a hash) to the beginning of below line
TurnOn: HTTPServletTracing