Gemfire: Cluster startup fails with "java.io.IOException: Keystore was tampered with, or password was incorrect"

Products

• VMware Tanzu Data Suite

Issue/Introduction

Gemfire cluster fails to startup with the the exception stacktrace below.

java.io.IOException: Keystore was tampered with, or password was incorrect
        at java.base/sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:813)
        at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:221)
        at java.base/java.security.KeyStore.load(KeyStore.java:1473)
        ............
        ............
     
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
        at gemfire//org.apache.geode.cache.internal.execute.FunctionToFileTracker.newFunction(FunctionToFileTracker.java:211)
        at gemfire//org.apache.geode.cache.internal.execute.FunctionToFileTracker.getRegisterableFunctionsFromClass(FunctionToFileTracker.java:194)
        at gemfire//org.apache.geode.cache.internal.execute.FunctionToFileTracker.loadFunctionFromClassName(FunctionToFileTracker.java:117)
        at gemfire//org.apache.geode.cache.internal.execute.FunctionToFileTracker.registerFunctionsFromFile(FunctionToFileTracker.java:77)
        at gemfire//com.vmware.gemfire.deployment.modules.internal.ModularJarDeploymentService.registerFunctions(ModularJarDeploymentService.java:205)
        at gemfire//com.vmware.gemfire.deployment.modules.internal.ModularJarDeploymentService.createModule(ModularJarDeploymentService.java:143)
        at gemfire//com.vmware.gemfire.deployment.modules.internal.ModularJarDeploymentService.updateModule(ModularJarDeploymentService.java:123)
        at gemfire//com.vmware.gemfire.deployment.modules.internal.ModularJarDeploymentService.deploy(ModularJarDeploymentService.java:100)
        at gemfire//com.vmware.gemfire.deployment.modules.internal.ModularJarDeploymentService.deploy(ModularJarDeploymentService.java:222)
        at gemfire//org.apache.geode.internal.cache.ClusterConfigurationLoader.deployJarsReceivedFromClusterConfiguration(ClusterConfigurationLoader.java:116)
        at gemfire//org.apache.geode.internal.cache.GemFireCacheImpl.initialize(GemFireCacheImpl.java:1436)
        at gemfire//org.apache.geode.internal.cache.InternalCacheBuilder.create(InternalCacheBuilder.java:198)
        at gemfire//org.apache.geode.internal.cache.InternalCacheBuilder.create(InternalCacheBuilder.java:163)
        at gemfire//org.apache.geode.cache.CacheFactory.create(CacheFactory.java:147)
        at gemfire//org.apache.geode.distributed.internal.DefaultServerLauncherCacheProvider.createCache(DefaultServerLauncherCacheProvider.java:56)
        at gemfire//org.apache.geode.distributed.ServerLauncher.createCache(ServerLauncher.java:927)
        at gemfire//org.apache.geode.distributed.ServerLauncher.start(ServerLauncher.java:828)
         ...
        .....
        at gemfire//com.vmware.gemfire.deployment.modules.internal.Launcher.invokeMainClass(Launcher.java:99)
        at gemfire//com.vmware.gemfire.deployment.modules.internal.Launcher.launch(Launcher.java:88)
        at gemfire//com.vmware.gemfire.deployment.modules.internal.Launcher.main(Launcher.java:103)
        at org.jboss.modules.Module.run(Module.java:353)
        at org.jboss.modules.Module.run(Module.java:321)
        at org.jboss.modules.Main.main(Main.java:604)
        at com.vmware.gemfire.bootstrap.internal.Launcher.launch(Launcher.java:64)
        at com.vmware.gemfire.bootstrap.Main.main(Main.java:27)
Caused by: java.security.UnrecoverableKeyException: Password verification failed
        at java.base/sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:811)
        ... 38 more

Resolution

The VMware Tanzu GemFire startup failure, indicated by java.io.IOException: Keystore was tampered with, or password was incorrect, is caused by a mismatch between the configured SSL password and the actual keystore file password.

1. Verify Password via Keytool: Run the following command manually to ensure you have the correct password for the file:

   keytool -list -v -keystore <your-keystore-file>

   Enter the password you have configured in your properties file.

2. If this fails, ensure that the following properties in your gfsecurity.properties /gemfire.properties or your start locator / start server command are correct:

   • ssl-keystore-password
   • ssl-truststore-password