User Threads remain as Zombies even after Server was Shut Down

Products

• VMware Tanzu Gemfire

Issue/Introduction

Symptoms:

Some user threads/process which are triggered from a cacheserver, such as a cachelistener or a SpringContextBootstrappingInitializer by the ThreadPoolTaskExecutor/ThreadPoolTaskScheduler, may remain in the JVM as a zombie process even after the cacheserver process was shut down or was forced out of the Gemfire cluster by some member-timeout mechanism.

Here is a thread dump example of a zombie process:

Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.80-b11 mixed mode):

"scheduler-97" prio=10 tid=0x0000000001ceb000 nid=0x2bf5 waiting on condition [0x00007f36efd2f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Locked ownable synchronizers:
- None
......
"ThreadPoolTaskExecutor-3" prio=10 tid=0x00007f36f9093800 nid=0x75d8 waiting on condition [0x00007f36f1c70000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Locked ownable synchronizers:
- None
......

Environment

Cause

Non-daemon threads running the user process, which were triggered by the GemFire cacheserver, may not exit from the JVM since they are non-daemon threads.

Resolution

Use a daemon thread instead of a non-daemon thread when creating a user thread/process by the ThreadPoolTaskExecutor/ThreadPoolTaskScheduler.

Example:

<bean id="scheduler" class="org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler"> 
 <property name="poolSize" value="100"/> 
 ...... 
 <property name="daemon" value="true"/> 
</bean> 

<bean id="ThreadPoolTaskExecutor" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor"> 
 <property name="maxPoolSize" value="20" /> 
 <property name="queueCapacity" value="10000" /> 
 ...... 
 <property name="daemon" value="true"/> 
</bean>