OneClick WebApp Occasionally Hangs and Doesn't Fully Launch - deadlock

Products

• CA Spectrum
• DX NetOps

Issue/Introduction

We've noticed that our Spectrum users who use WebApp to launch OneClick sometimes can't get it to launch.  The only solution is to kill the 2 webtomcat processes and then
    restart it.  A few minutes later, WebApp is then functional.  This problem happens periodically once every 1 - 2 weeks.

Environment

Release : 22.2

Cause

This was due to a bug in the 3rd party Webswing application whereby a deadlock condition can occur that prevents the Webtomcat from working and leaving users
   unable to launch the WebApp.

Generating java stack dumps against the webtomcat process shows the deadlock.

Analysis of DeadLock

Resolution

 

Fixed in Spectrum 22.2.7 which ships WebSwing 22.2.5 which resolves the periodic deadlock issue. 

 

Spectrum 22.2.2-4 ships with WebSwing 22.1.4, however, this is fixed in 22.1.10 and newer

The WebSwing deadlock has been fixed in version 22.1.10
See issue #755: https://www.webswing.org/docs/22.1/start/news.html#22110-lts

 

Additional Information

 

To diagnose the hang one needs to generate Java Stack dumps by running (linux) kill -3 against the webtomcat pid.

Note: We do not want the xvfb-run process, we need the PID of the  java process

example:
     kill -3 1834

We will also need 4 - 5 stack dumps about 60 seconds apart. This will show if the threads are in the same spot. In this case, we were able to 
    see  threads blocked and find one of the threads has holding a lock that another was waiting for while this thread was waiting for the lock
    the other had caused the deadlock.