VMware GemFire Cacheserver process failed to exit due to non-daemon thread

Products

• VMware Tanzu Gemfire

Issue/Introduction

Customer is using SpringContextBootstrappingInitializer which is defined in cache.xml's initializer to call ThreadPoolTaskExecutor/ThreadPoolTaskScheduler bean to run tasks like loading data from external Database into VMware GemFire when cacheserver is starting.

Current problem is that some zombie threads which is created by ThreadPoolTaskExecutor/ThreadPoolTaskScheduler will remain in the Cacheserver jvm process even VMware GemFire related threads are closed completely when Gemfire Cacheserver is shutting down.

• initializer's sample setting in Cache.xml

<initializer>
		<class-name>org.springframework.data.gemfire.support.SpringContextBootstrappingInitializer</class-name>
		<parameter name="contextConfigLocations">
			<string>
				classpath:config/customer-gemfire-dbaccess.xml,
				classpath:config/customer-gemfire-server-beans.xml
			</string>
		</parameter>
</initializer>

• ThreadPoolTaskExecutor bean sample setting:

	<bean id="ThreadPoolTaskExecutor"
		class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
		<property name="corePoolSize" value="8"></property>
		<property name="keepAliveSeconds" value="3000" />
		<property name="maxPoolSize" value="10" />
		<property name="queueCapacity" value="10000" />
	</bean>

• ThreadPoolTaskScheduler bean sample setting: 

<bean id="scheduler" class="org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler">
 <property name="poolSize" value="100"/>
</bean>

Environment

Product Version: 9.0

Resolution

From the below thread dump, you can see that the zombie threads are all non-daemon threads, this is the reason why Cacheserver process will not exit.

"scheduler-97" prio=10 tid=0x0000000001ceb000 nid=0x2bf5 waiting on condition [0x00007f36efd2f000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000079279a8f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)


"ThreadPoolTaskExecutor-3" prio=10 tid=0x00007f36f9093800 nid=0x75d8 waiting on condition [0x00007f36f1c70000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000007927b2240> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

The resolution for this issue is to add <property name="daemon" value="true"/> property to the ThreadPoolTaskScheduler ThreadPoolTaskExecutorbean so that the created threads are all daemon thread and Cacheserver process will exit gracefully without any zombie non-daemon threads.

For example:

<bean id="scheduler" class="org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler">
 <property name="poolSize" value="100"/>
 <property name="daemon" value="true"/>
</bean>