OneClick WebApp Occasionally Hangs and Doesn't Fully Launch - deadlock
Article ID: 259589
Updated On:
Products
Issue/Introduction
We've noticed that our Spectrum users who use WebApp to launch OneClick sometimes can't get it to launch. The only solution is to kill the 2 webtomcat processes and then
restart it. A few minutes later, WebApp is then functional. This problem happens periodically once every 1 - 2 weeks.
Environment
Release : 22.2
Cause
This was due to a bug in the 3rd party Webswing application whereby a deadlock condition can occur that prevents the Webtomcat from working and leaving users
unable to launch the WebApp.
Generating java stack dumps against the webtomcat process shows the deadlock.
Analysis of DeadLock
Resolution
Fixed in Spectrum 22.2.7 which ships WebSwing 22.2.5 which resolves the periodic deadlock issue.
Spectrum 22.2.2-4 ships with WebSwing 22.2.4, however, this is fixed in 22.1.10 and newer
The WebSwing deadlock has been fixed in version 22.1.10
See issue #755: https://www.webswing.org/docs/22.1/start/news.html#22110-lts
Additional Information
To diagnose the hang one needs to generate Java Stack dumps by running (linux) kill -3 against the webtomcat pid.
Note: We do not want the xvfb-run process, we need the PID of the java process
example:
kill -3 1834
this is output to standard output which for webtomcat is
<$SPECROOT>/webtomcat/logs/catalina.out
We will also need 4 - 5 stack dumps about 60 seconds apart. This will show if the threads are in the same spot. In this case, we were able to
see threads blocked and find one of the threads has holding a lock that another was waiting for while this thread was waiting for the lock
the other had caused the deadlock.
The procedure for this is as a root user in a bash shell please enter this command
pstack <webtomcat Pid> >> pstack.out
Please run this 5 times in 1 minute increments, once this has been completed please provide the pstack.out file and the <$SPECROOT>/webtomcat/logs/catalina.out to support.
Feedback
