Client Application running on kubernetes container cannot connect to GemFire server

Products

• VMware Tanzu Gemfire

Issue/Introduction

Client applications running on kubernetes cluster failed to connect to the GemFire cluster, which is outside of the kubernetes cluster and running on the bare-metal host due to a "org.apache.geode.InternalGemFireError: Cannot resolve local host name to an IP address" error while other client applications running on a bare-metal host can successfully connect to the GemFire cluster.

Failed client application log with exception stack:

2020-09-04T02:15:05.429426599Z 2020-09-04 12:15:05.429  WARN 1 --- [           main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'exampleServiceRunner': Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'exampleService': Invocation of init method failed; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'cache-geode': Unsatisfied dependency expressed through field 'gemfireTemplate'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'regionFactory' defined in class path resource [com/example/config/GeodeConfiguration.class]: Unsatisfied dependency expressed through method 'regionFactory' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gemfireCache': FactoryBean threw exception on object creation; nested exception is org.apache.geode.InternalGemFireError: Cannot resolve local host name to an IP address
2020-09-04T02:15:05.429612579Z 2020-09-04 12:15:05.429  WARN 1 --- [           main] o.s.b.f.support.DisposableBeanAdapter    : Invocation of destroy method failed on bean with name 'gemfireCache': org.apache.geode.cache.CacheClosedException: A cache has not yet been created.
......
2020-09-04T02:15:13.118285519Z org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'exampleServiceRunner': Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'exampleService': Invocation of init method failed; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'cache-geode': Unsatisfied dependency expressed through field 'gemfireTemplate'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'regionFactory' defined in class path resource [com/example/config/GeodeConfiguration.class]: Unsatisfied dependency expressed through method 'regionFactory' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gemfireCache': FactoryBean threw exception on object creation; nested exception is org.apache.geode.InternalGemFireError: Cannot resolve local host name to an IP address
2020-09-04T02:15:13.118303329Z 	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:797) ~[spring-beans-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
2020-09-04T02:15:13.118307539Z 	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:227) ~[spring-beans-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
2020-09-04T02:15:13.118311015Z 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1356) ~[spring-beans-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
2020-09-04T02:15:13.118314503Z 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1203) ~[spring-beans-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
2020-09-04T02:15:13.11831868Z 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) ~[spring-beans-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
2020-09-04T02:15:13.118322249Z 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) ~[spring-beans-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
2020-09-04T02:15:13.118325629Z 	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) ~[spring-beans-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
2020-09-04T02:15:13.11832902Z 	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226) ~[spring-beans-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
2020-09-04T02:15:13.118332389Z 	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) ~[spring-beans-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
2020-09-04T02:15:13.118335659Z 	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
2020-09-04T02:15:13.118338957Z 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:897) ~[spring-beans-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
2020-09-04T02:15:13.118342377Z 	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:879) ~[spring-context-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
2020-09-04T02:15:13.118345792Z 	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:551) ~[spring-context-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
2020-09-04T02:15:13.118351645Z 	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:143) ~[spring-boot-2.3.2.RELEASE.jar!/:2.3.2.RELEASE]
2020-09-04T02:15:13.118355152Z 	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:758) ~[spring-boot-2.3.2.RELEASE.jar!/:2.3.2.RELEASE]
2020-09-04T02:15:13.11835842Z 	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:750) ~[spring-boot-2.3.2.RELEASE.jar!/:2.3.2.RELEASE]
2020-09-04T02:15:13.118361924Z 	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397) ~[spring-boot-2.3.2.RELEASE.jar!/:2.3.2.RELEASE]
2020-09-04T02:15:13.118367184Z 	at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) ~[spring-boot-2.3.2.RELEASE.jar!/:2.3.2.RELEASE]
2020-09-04T02:15:13.118370574Z 	at com.example.exampleServiceRunner.main(exampleServiceRunner.java:95) ~[classes!/:na]
......
2020-09-04T02:15:13.118611211Z Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gemfireCache': FactoryBean threw exception on object creation; nested exception is org.apache.geode.InternalGemFireError: Cannot resolve local host name to an IP address
2020-09-04T02:15:13.118614766Z 	at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:176) ~[spring-beans-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
2020-09-04T02:15:13.11861812Z 	at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:101) ~[spring-beans-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
2020-09-04T02:15:13.118621476Z 	at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1827) ~[spring-beans-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
2020-09-04T02:15:13.118624894Z 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.getObjectForBeanInstance(AbstractAutowireCapableBeanFactory.java:1265) ~[spring-beans-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
2020-09-04T02:15:13.118628244Z 	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:334) ~[spring-beans-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
2020-09-04T02:15:13.118631534Z 	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
2020-09-04T02:15:13.118634823Z 	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) ~[spring-beans-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
2020-09-04T02:15:13.118638117Z 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307) ~[spring-beans-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
2020-09-04T02:15:13.118641415Z 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227) ~[spring-beans-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
2020-09-04T02:15:13.118644762Z 	at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884) ~[spring-beans-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
2020-09-04T02:15:13.118648212Z 	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788) ~[spring-beans-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
2020-09-04T02:15:13.118651574Z 	... 74 common frames omitted
2020-09-04T02:15:13.118656083Z Caused by: org.apache.geode.InternalGemFireError: Cannot resolve local host name to an IP address
2020-09-04T02:15:13.11865942Z 	at org.apache.geode.distributed.internal.LonerDistributionManager.generateMemberId(LonerDistributionManager.java:1222) ~[geode-core-1.12.0.jar!/:na]
2020-09-04T02:15:13.118662727Z 	at org.apache.geode.distributed.internal.LonerDistributionManager.<init>(LonerDistributionManager.java:91) ~[geode-core-1.12.0.jar!/:na]
2020-09-04T02:15:13.118666163Z 	at org.apache.geode.distributed.internal.InternalDistributedSystem.initialize(InternalDistributedSystem.java:794) ~[geode-core-1.12.0.jar!/:na]
2020-09-04T02:15:13.118669435Z 	at org.apache.geode.distributed.internal.InternalDistributedSystem.access$200(InternalDistributedSystem.java:135) ~[geode-core-1.12.0.jar!/:na]
2020-09-04T02:15:13.118672718Z 	at org.apache.geode.distributed.internal.InternalDistributedSystem$Builder.build(InternalDistributedSystem.java:3036) ~[geode-core-1.12.0.jar!/:na]
2020-09-04T02:15:13.118675978Z 	at org.apache.geode.distributed.internal.InternalDistributedSystem.connectInternal(InternalDistributedSystem.java:290) ~[geode-core-1.12.0.jar!/:na]
2020-09-04T02:15:13.118679436Z 	at org.apache.geode.distributed.internal.InternalDistributedSystem.connectInternal(InternalDistributedSystem.java:216) ~[geode-core-1.12.0.jar!/:na]
2020-09-04T02:15:13.118684271Z 	at org.apache.geode.cache.client.ClientCacheFactory.connectInternalDistributedSystem(ClientCacheFactory.java:280) ~[geode-core-1.12.0.jar!/:na]
2020-09-04T02:15:13.118689017Z 	at org.apache.geode.cache.client.ClientCacheFactory.basicCreate(ClientCacheFactory.java:250) ~[geode-core-1.12.0.jar!/:na]
2020-09-04T02:15:13.118693377Z 	at org.apache.geode.cache.client.ClientCacheFactory.create(ClientCacheFactory.java:216) ~[geode-core-1.12.0.jar!/:na]
2020-09-04T02:15:13.118697832Z 	at org.springframework.data.gemfire.client.ClientCacheFactoryBean.createCache(ClientCacheFactoryBean.java:397) ~[spring-data-geode-2.3.2.RELEASE.jar!/:2.3.2.RELEASE]
2020-09-04T02:15:13.118702444Z 	at org.springframework.data.gemfire.CacheFactoryBean.resolveCache(CacheFactoryBean.java:321) ~[spring-data-geode-2.3.2.RELEASE.jar!/:2.3.2.RELEASE]
2020-09-04T02:15:13.118707099Z 	at org.springframework.data.gemfire.CacheFactoryBean.init(CacheFactoryBean.java:267) ~[spring-data-geode-2.3.2.RELEASE.jar!/:2.3.2.RELEASE]
2020-09-04T02:15:13.118711677Z 	at java.util.Optional.orElseGet(Optional.java:267) ~[na:1.8.0_221]
2020-09-04T02:15:13.118716345Z 	at org.springframework.data.gemfire.CacheFactoryBean.getObject(CacheFactoryBean.java:774) ~[spring-data-geode-2.3.2.RELEASE.jar!/:2.3.2.RELEASE]
2020-09-04T02:15:13.118721284Z 	at org.springframework.data.gemfire.CacheFactoryBean.getObject(CacheFactoryBean.java:108) ~[spring-data-geode-2.3.2.RELEASE.jar!/:2.3.2.RELEASE]
2020-09-04T02:15:13.118725384Z 	at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:169) ~[spring-beans-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
2020-09-04T02:15:13.118728768Z 	... 84 common frames omitted

Locator log when failed client application tried to connect:

[info 2020/09/04 12:15:25.131 AEST <locator request thread 998> tid=0x11e2] Exception in processing request from 192.###.#.##
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1002)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
at org.apache.geode.internal.net.SocketCreator.handshakeIfSocketIsSSL(SocketCreator.java:814)
at org.apache.geode.distributed.internal.tcpserver.TcpServer.lambda$processRequest$0(TcpServer.java:307)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.read(InputRecord.java:505)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
... 8 more

Environment

Product Version: 9.10

Resolution

According to the above exception stack and symptom, the issue could be caused by the client application running in the kubernetes' container failing to resolve the GemFire locator's IP address.

After confirmation, the client application's container's base image is using an alpine Linux image which is lacking  /etc/nsswitch.conf.

This issue is resolved by adding /etc/nsswitch.conf to the alpine Linux base image where nsswitch.conf allows the client application to find the hostname/IP address using reverse lookup from local /etc/hosts file.

Example of nsswitch.conf:

hosts: files dns