GemFire servers intermittently hit an exception, during close cache operation

Products

• VMware Tanzu Gemfire

Issue/Introduction

Occasionally, GemFire servers hit the following exception, while stopping or shutting down the cluster using the "gfsh" commands.

[warn xxxx/yy/zz aa:bb:ccc.ddd ABC eeee-ffff-gggg.server <Shutdown member> tid=0x12345] Exception trying to close cache
org.apache.geode.deployment.internal.exception.ServiceLoadingFailureException: No implementation of JarDeploymentService could be loaded.
        at gemfire//org.apache.geode.deployment.internal.DeploymentServiceFactory.createJarDeploymentService(DeploymentServiceFactory.java:47)
        at gemfire//org.apache.geode.deployment.internal.DeploymentServiceFactory.getJarDeploymentServiceInstance(DeploymentServiceFactory.java:69)
        at gemfire//org.apache.geode.classloader.internal.ClassPathLoader.<init>(ClassPathLoader.java:86)
        at gemfire//org.apache.geode.classloader.internal.ClassPathLoader.close(ClassPathLoader.java:132)
        at gemfire//org.apache.geode.internal.cache.GemFireCacheImpl.doClose(GemFireCacheImpl.java:2537)
        at gemfire//org.apache.geode.internal.cache.GemFireCacheImpl.close(GemFireCacheImpl.java:2218)
        at gemfire//org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:1555)
        at gemfire//org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:1246)
        at gemfire//org.apache.geode.management.internal.beans.MemberMBeanBridge.lambda$shutDownMember$1(MemberMBeanBridge.java:788)
        at java.base/java.lang.Thread.run(Thread.java:829)

Environment

GemFire Version: 10.1.0

Cause

There are potentially two issues here.

(1) If any processes are hanging around even after the shutdown, then this is due to some user's thread remaining active after stopping or shutting down the cluster. This is not a GemFire issue. The GemFire threads will get closed when the cache is closed (server shutdown).

(2) Exception with “No implementation of JarDeploymentService” - Warning message, can be ignored. This is happening in the JDK/java code.

Resolution

If any processes are hanging around even after the shutdown, then the root cause is user’s query must have initiated a long running thread.This thread caused the server to become zombie when server is stopped. The issue needs to be investigated from the application thread perspective. Please check, if there are any long-running user threads and take corrective action.

To make sure that the server restart is successful, user need to either remove the server directory’s vf.gf.server.pid or kill the zombie process. We recommend to do both.