Tomcat startup hangs and the tomcat log notes Tomcat not started yet over and over

Products

• Spectrum
• Network Observability

Issue/Introduction

Tomcat hangs on startup.  In the tomcat log you can see:

 

Oct 19, 2022 16:59:33.373 (PoolThread-1: GlobalPool => ClientLogPurgeCheckerTask) (GAPM) - GAPM.user group count : 32
Oct 19, 2022 16:59:38.385 (PoolThread-1: GlobalPool => ClientLogPurgeCheckerTask) (GAPM) - GAPM.finished processing user groups watch registration in : 5012 ms
Oct 19, 2022 16:59:38.385 (PoolThread-1: GlobalPool => ClientLogPurgeCheckerTask) (GAPM) - GAPM.users count : 3148
Oct 19, 2022 17:03:30.470 (PoolThread-1: GlobalPool => ClientLogPurgeCheckerTask) (GAPM) - GAPM.finished processing user watch registration in : 232085 ms
Oct 19, 2022 17:03:30.487 (NETQOS/Startup/Container) (ALARMS) - Alarm fire timer scheduled at: 10000
Oct 19, 2022 17:03:30.488 (NETQOS/Startup/Container) (ALARMS) - alarm.web.model.gatherpoolsize : 5
Oct 19, 2022 17:03:37.007 - ServiceDeskIntegrationManager - Tomcat Startup Monitor with settings: Initial Wait: 0 , Check Frequency: 5
Oct 19, 2022 17:03:37.008 - WARNING - ServiceDeskIntegrationManager : Tomcat not started yet
Oct 19, 2022 17:03:42.008 - WARNING - ServiceDeskIntegrationManager : Tomcat not started yet
Oct 19, 2022 17:03:47.010 - WARNING - ServiceDeskIntegrationManager : Tomcat not started yet
Oct 19, 2022 17:03:52.010 - WARNING - ServiceDeskIntegrationManager : Tomcat not started yet
Oct 19, 2022 17:03:57.011 - WARNING - ServiceDeskIntegrationManager : Tomcat not started yet
Oct 19, 2022 17:04:02.012 - WARNING - ServiceDeskIntegrationManager : Tomcat not started yet
Oct 19, 2022 17:04:07.013 - WARNING - ServiceDeskIntegrationManager : Tomcat not started yet
Oct 19, 2022 17:04:12.014 - WARNING - ServiceDeskIntegrationManager : Tomcat not started yet
Oct 19, 2022 17:04:17.015 - WARNING - ServiceDeskIntegrationManager : Tomcat not started yet
Oct 19, 2022 17:04:22.016 - WARNING - ServiceDeskIntegrationManager : Tomcat not started yet
Oct 19, 2022 17:04:27.018 - WARNING - ServiceDeskIntegrationManager : Tomcat not started yet

Half hour later it's still not started:

Oct 19, 2022 17:26:52.320 - WARNING - ServiceDeskIntegrationManager : Tomcat not started yet
Oct 19, 2022 17:26:57.321 - WARNING - ServiceDeskIntegrationManager : Tomcat not started yet
Oct 19, 2022 17:27:02.321 - WARNING - ServiceDeskIntegrationManager : Tomcat not started yet
Oct 19, 2022 17:27:07.323 - WARNING - ServiceDeskIntegrationManager : Tomcat not started yet
Oct 19, 2022 17:27:12.325 - WARNING - ServiceDeskIntegrationManager : Tomcat not started yet
Oct 19, 2022 17:27:17.325 - WARNING - ServiceDeskIntegrationManager : Tomcat not started yet
Oct 19, 2022 17:27:22.327 - WARNING - ServiceDeskIntegrationManager : Tomcat not started yet

Kill -3 on the OC tomcat process shows NetQoS processing:

"NETQOS/Startup/Container" #342 prio=5 os_prio=0 tid=0x00007f99d9ec8000 nid=0x9bbb waiting on condition [0x00007f9924eb2000]
   java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for  <0x00007f9fe84a17a8> (a java.util.concurrent.CountDownLatch$Sync)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
 at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
 at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
 at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
 at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
 at com.aprisma.spectrum.app.topo.web.repository.ModelRepository.waitForReadTasks(ModelRepository.java:1895)

Environment

Release : 21.2

Cause

Connectivity to the SpectroSERVERs

Resolution

We believe the problem was due to network latencies and connection issues due to MTU sizes however we never really nailed this down.

If this happens again, the debug instructions are:

1.  Edit the $SPECROOT/tomcat/webapps/spectrum/WEB-INF/web.xml file and search for the following lines:

<param-value>NetqosEventLog@[e-mail]@off;
NetqosAlarmSyncTrackingLog@[e-mail]acking@off;
NetqosSyncLog@[e-mail]@off;</param-value>

       Change the value to @on for all 3 

   2. Start Tomcat and wait 30 minutes for the hang to happen

   3. Dump the threads using ./kill -3 OC_pid for 5 times 1 minute apart

------------------------------------

Instructions if machine is running on windows:

1- Go to $SPECROOT\lib\sdpm\ directory;

2- Run the following command:

kill -break <OC tomcat process PID>

3- The thread dump info will be recorded in tomcat\logs\jvm.log file

Additional Information

Contact Support with the files created for further assistance.