PCC service instance or GemFire server not starting due to NullPointerException in server logs

Products

• VMware Tanzu Gemfire

Issue/Introduction

This issue can come up in a PCC service instance or a GemFire cluster.

In the PCC service instance:
You may see that the gemfire-server job keeps failing on the PCC service instance VMs. The gemfire logs (server.log or one of the rolled over logs) located in /var/vcap/sys/log/gemfire-server/gemfire will show the below mentioned exception. 

In a GemFire Cluster:
Then GemFire cache server will not come up on startup and you will notice the below exception in the cache server logs.
 

Exception in thread "main" java.lang.NullPointerException
	at org.apache.geode.internal.InternalDataSerializer.readPdxSerializable(InternalDataSerializer.java:3254)
	at org.apache.geode.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:3069)
	at org.apache.geode.DataSerializer.readObject(DataSerializer.java:2978)
	at org.apache.geode.internal.util.BlobHelper.deserializeBlob(BlobHelper.java:90)
	at org.apache.geode.internal.cache.EntryEventImpl.deserialize(EntryEventImpl.java:1979)
	at org.apache.geode.internal.cache.Oplog.deserializeKey(Oplog.java:6335)
	at org.apache.geode.internal.cache.Oplog.readKrf(Oplog.java:1740)
	at org.apache.geode.internal.cache.Oplog.recoverCrf(Oplog.java:2209)
	at org.apache.geode.internal.cache.PersistentOplogSet.recoverOplogs(PersistentOplogSet.java:456)
	at org.apache.geode.internal.cache.PersistentOplogSet.recoverRegionsThatAreReady(PersistentOplogSet.java:369)
	at org.apache.geode.internal.cache.DiskStoreImpl.recoverRegionsThatAreReady(DiskStoreImpl.java:2069)
	at org.apache.geode.internal.cache.DiskStoreImpl.initializeIfNeeded(DiskStoreImpl.java:2057)
	at org.apache.geode.internal.cache.DiskStoreImpl.doInitialRecovery(DiskStoreImpl.java:2062)
	at org.apache.geode.internal.cache.DiskStoreFactoryImpl.initializeDiskStore(DiskStoreFactoryImpl.java:178)
	at org.apache.geode.internal.cache.DiskStoreFactoryImpl.create(DiskStoreFactoryImpl.java:144)
	at org.apache.geode.internal.cache.GemFireCacheImpl.getOrCreateDefaultDiskStore(GemFireCacheImpl.java:2592)
	at org.apache.geode.internal.cache.xmlcache.CacheCreation.initializePdxDiskStore(CacheCreation.java:775)
	at org.apache.geode.internal.cache.xmlcache.CacheCreation.create(CacheCreation.java:501)
	at org.apache.geode.internal.cache.xmlcache.CacheXmlParser.create(CacheXmlParser.java:337)
	at org.apache.geode.internal.cache.GemFireCacheImpl.loadCacheXml(GemFireCacheImpl.java:4308)
	at org.apache.geode.internal.cache.ClusterConfigurationLoader.applyClusterXmlConfiguration(ClusterConfigurationLoader.java:195)
	at org.apache.geode.internal.cache.GemFireCacheImpl.applyJarAndXmlFromClusterConfig(GemFireCacheImpl.java:1261)
	at org.apache.geode.internal.cache.GemFireCacheImpl.initialize(GemFireCacheImpl.java:1225)
	at org.apache.geode.internal.cache.GemFireCacheImpl.basicCreate(GemFireCacheImpl.java:793)
	at org.apache.geode.internal.cache.GemFireCacheImpl.create(GemFireCacheImpl.java:779)
	at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:177)
	at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:224)
	at org.apache.geode.distributed.internal.DefaultServerLauncherCacheProvider.createCache(DefaultServerLauncherCacheProvider.java:52)
	at org.apache.geode.distributed.ServerLauncher.createCache(ServerLauncher.java:856)
	at org.apache.geode.distributed.ServerLauncher.start(ServerLauncher.java:774)
	at org.apache.geode.distributed.ServerLauncher.run(ServerLauncher.java:704)
	at org.apache.geode.distributed.ServerLauncher.main(ServerLauncher.java:224)

Resolution

In GemFire or PCC you can have client applications serialize data using PDX Serialization before sending data to the server side.

There is metadata w.r.t PDX stored on the servers. This metadata is called PDXTypes. 
GemFire does not support having PDX serialized keys. The exception mentioned above occurs when you have PDX serialized keys sitting on the same disk-store as the PDXTypes. The only way to get the server backup is to delete the disk-stores but that has a potential for data loss.

The fix is not to use domain objects which are PDX serialized as keys and that is the best practice as described here. 

On the client app side, when using Spring-data-gemfire or spring-boot-data-geode the @EnablePdx annotation makes your domain objects PDX serialized when sending to server. So it might not be apparent that your keys are being PDX serialized. Best practice is to just use primitives or wrapped primitives (String, Integer) as described here.