ClassCastException when running deployed GemFire functions
Article ID: 294342
Updated On:
Products
VMware Tanzu Gemfire
Issue/Introduction
When you run a GemFire function from a client after you deploy jars, including the function execution service and domain classes in the GemFire cluster, you may see the following ClassCastException on the client side:
org.apache.geode.cache.execute.FunctionException: org.apache.geode.cache.client.ServerOperationException: remote server on cacheserver3(1958:loner):51558:811a2b45: While performing a remote executeRegionFunctionSingleHop at org.apache.geode.internal.cache.execute.ServerRegionFunctionExecutor.executeOnServer(ServerRegionFunctionExecutor.java:229) at org.apache.geode.internal.cache.execute.ServerRegionFunctionExecutor.executeFunction(ServerRegionFunctionExecutor.java:180) at org.apache.geode.internal.cache.execute.ServerRegionFunctionExecutor.execute(ServerRegionFunctionExecutor.java:379) at com.sample.gemfire.position.repositories.PositionGemFireRepository.lambda$getPositions$3(PositionGemFireRepository.java:157) at com.sample.gemfire.LogHelper.withMDC(LogHelper.java:14) at com.sample.gemfire.position.repositories.PositionGemFireRepository.getPositions(PositionGemFireRepository.java:123) at com.sample.gemfire.position.repositories.PositionGemFireRepository.getAccountBalances(PositionGemFireRepository.java:323) ...... at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.geode.cache.client.ServerOperationException: remote server on cacheserver3(1958:loner):51558:811a2b45: While performing a remote executeRegionFunctionSingleHop at org.apache.geode.cache.client.internal.ExecuteRegionFunctionSingleHopOp$ExecuteRegionFunctionSingleHopOpImpl.processResponse(ExecuteRegionFunctionSingleHopOp.java:372) at org.apache.geode.cache.client.internal.AbstractOp.processResponse(AbstractOp.java:225) at org.apache.geode.cache.client.internal.AbstractOp.attemptReadResponse(AbstractOp.java:198) at org.apache.geode.cache.client.internal.AbstractOp.attempt(AbstractOp.java:386) at org.apache.geode.cache.client.internal.ConnectionImpl.execute(ConnectionImpl.java:269) at org.apache.geode.cache.client.internal.pooling.PooledConnection.execute(PooledConnection.java:325) at org.apache.geode.cache.client.internal.OpExecutorImpl.executeWithPossibleReAuthentication(OpExecutorImpl.java:892) at org.apache.geode.cache.client.internal.OpExecutorImpl.executeOnServer(OpExecutorImpl.java:387) at org.apache.geode.cache.client.internal.OpExecutorImpl.executeOn(OpExecutorImpl.java:349) at org.apache.geode.cache.client.internal.PoolImpl.executeOn(PoolImpl.java:798) at org.apache.geode.cache.client.internal.SingleHopOperationCallable.call(SingleHopOperationCallable.java:51) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted Caused by: java.lang.ClassCastException: com.sample.api.criteria.position.PositionCriteria cannot be cast to com.sample.api.criteria.ValidateArgument at com.sample.gemfire.functions.FinDataAwareFunction.execute(FinDataAwareFunction.java:59) at org.apache.geode.internal.cache.execute.AbstractExecution.executeFunctionLocally(AbstractExecution.java:331) at org.apache.geode.internal.cache.execute.AbstractExecution$1.run(AbstractExecution.java:272) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.geode.distributed.internal.ClusterDistributionManager.runUntilShutdown(ClusterDistributionManager.java:952) at org.apache.geode.distributed.internal.ClusterDistributionManager.doFunctionExecutionThread(ClusterDistributionManager.java:806) at org.apache.geode.internal.logging.LoggingThreadFactory.lambda$newThread$0(LoggingThreadFactory.java:121) ... 1 common frames omitted
Environment
Product Version: 9.7
Resolution
This issues is related to how GemFire loads classes. Before GemFire 9.9, when a jar is deployed, a new class loader is created and it doesn't chain together them - GemFire keeps the deserialized bits around for performance consideration.
This can be managed by the system property: -Dgemfire.loadClassOnEveryDeserialization. When the data model is in a jar and that jar gets updated, any previously deserialized bits are the same type as in the old jar, even if the class doesn't change in the new jar. As a result, the ClassCastException is thrown.
To workaround this issue, separate the domain model from functions to reduce the chance of this happening. Include the domain model classes in a jar and put them in CLASSPATH. Deploy the function jars into GemFire cluster.
GemFire 9.9 improves the way jars are deployed and the way classes are loaded - it can resolve this ClassCastException issue. More details can be found here: GEODE-6822 - Deploying a jar causes a new class loader to be created, cause possible mismatch.
This can be managed by the system property: -Dgemfire.loadClassOnEveryDeserialization. When the data model is in a jar and that jar gets updated, any previously deserialized bits are the same type as in the old jar, even if the class doesn't change in the new jar. As a result, the ClassCastException is thrown.
To workaround this issue, separate the domain model from functions to reduce the chance of this happening. Include the domain model classes in a jar and put them in CLASSPATH. Deploy the function jars into GemFire cluster.
GemFire 9.9 improves the way jars are deployed and the way classes are loaded - it can resolve this ClassCastException issue. More details can be found here: GEODE-6822 - Deploying a jar causes a new class loader to be created, cause possible mismatch.
Feedback
Yes
No
Powered by
