When using a CacheLoader, throwing a CacheLoaderException that contains nested objects that are not serializable will cause an uncaught exception that could result in a P2P response never getting sent and threads waiting indefinitely on the peer node that initiated the cache load. In such a situation, the node where the CacheLoader is executing would log messages like the following:
[severe 2018/02/15 18:15:14.808 EST cache1 :49000 unshared ordered uid=900 dom #1 port=4871> tid=0x751] Uncaught exception processing partitioned.GetMessage(prid=1 (name = "/loaderRegion") processorId=0; posDup=false; key=io.pivotal.supportTest.keyType; callback arg=null; context=identity(client(15:loner):49013:27b8539a,connection=1) com.gemstone.gemfire.InternalGemFireException: java.io.NotSerializableException: io.pivotal.supportTest.CustomLoaderError at com.gemstone.gemfire.internal.tcp.DirectReplySender.putOutgoing(DirectReplySender.java:76) at com.gemstone.gemfire.distributed.internal.ReplyMessage.send(ReplyMessage.java:109) at com.gemstone.gemfire.internal.cache.partitioned.PartitionMessage.sendReply(PartitionMessage.java:392) at com.gemstone.gemfire.internal.cache.partitioned.PartitionMessage.process(PartitionMessage.java:376) at com.gemstone.gemfire.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:386) at com.gemstone.gemfire.distributed.internal.DistributionMessage.schedule(DistributionMessage.java:449) at com.gemstone.gemfire.distributed.internal.DistributionManager.scheduleIncomingMessage(DistributionManager.java:3872) at com.gemstone.gemfire.distributed.internal.DistributionManager.handleIncomingDMsg(DistributionManager.java:3496) at com.gemstone.gemfire.distributed.internal.DistributionManager$MyListener.messageReceived(DistributionManager.java:4693) at com.gemstone.gemfire.distributed.internal.membership.jgroup.JGroupMembershipManager.processMessage(JGroupMembershipManager.java:2128) at com.gemstone.gemfire.distributed.internal.membership.jgroup.JGroupMembershipManager.handleOrDeferMessage(JGroupMembershipManager.java:2037) at com.gemstone.gemfire.distributed.internal.membership.jgroup.JGroupMembershipManager$MyDCReceiver.messageReceived(JGroupMembershipManager.java:647) at com.gemstone.gemfire.distributed.internal.direct.DirectChannel.receive(DirectChannel.java:804) at com.gemstone.gemfire.internal.tcp.TCPConduit.messageReceived(TCPConduit.java:835) at com.gemstone.gemfire.internal.tcp.Connection.dispatchMessage(Connection.java:3932) at com.gemstone.gemfire.internal.tcp.Connection.processNIOBuffer(Connection.java:3515) at com.gemstone.gemfire.internal.tcp.Connection.runNioReader(Connection.java:1827) at com.gemstone.gemfire.internal.tcp.Connection.run(Connection.java:1702) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.NotSerializableException: io.pivotal.supportTest.CustomLoaderError ...
The peer nodes would show an accumulation of ReplyWaitsInProgess in their statistics and possibly an accumulation of ServerConnection, or similar, threads.
On an exception, the CacheLoader's load()<\code> method throws CacheLoaderException, which is not pdx serializable but is simply Serializable. Hence, once DataSerializer.writeObject gets into the Serializable framework, it uses that for the entire object, so anything nest inside that CacheLoaderException will need to be Serializable and Pdx will not come into play.