Unable to create service in VMware Tanzu GemFire 1.6 and above
search cancel

Unable to create service in VMware Tanzu GemFire 1.6 and above

book

Article ID: 294311

calendar_today

Updated On:

Products

VMware Tanzu Gemfire

Issue/Introduction

Service instance creation fails in VMware Tanzu GemFire 1.6.0 and above. The locator logs show the following: 
[warning 2019/10/23 18:34:50.253 UTC locator-3df92bb6-816b-469d-a37f-dc4dc3c713b0 <ThreadsMonitor> tid=0x10] Thread <86> is stuck

[warning 2019/10/23 18:34:50.255 UTC locator-3df92bb6-816b-469d-a37f-dc4dc3c713b0 <ThreadsMonitor> tid=0x10] Thread <86> that was executed at <23 O
ct 2019 18:34:12 UTC> has been stuck for <37.643 seconds> and number of thread monitor iteration <1>
  Thread Name <Pooled High Priority Message Processor 2>
  Thread state <RUNNABLE>
  Executor Group <PooledExecutorWithDMStats>
  Monitored metric <ResourceManagerStats.numThreadsStuck>
  Thread Stack:
  sun.nio.ch.Net.poll(Native Method)
  sun.nio.ch.SocketChannelImpl.poll(SocketChannelImpl.java:954)
  sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:110)
  org.apache.geode.internal.tcp.Connection.<init>(Connection.java:1235)
  org.apache.geode.internal.tcp.Connection.createSender(Connection.java:1066)
  org.apache.geode.internal.tcp.ConnectionTable.handleNewPendingConnection(ConnectionTable.java:305)
  org.apache.geode.internal.tcp.ConnectionTable.getSharedConnection(ConnectionTable.java:413)
  org.apache.geode.internal.tcp.ConnectionTable.get(ConnectionTable.java:598)
  org.apache.geode.internal.tcp.TCPConduit.getConnection(TCPConduit.java:947)
  org.apache.geode.distributed.internal.direct.DirectChannel.getConnections(DirectChannel.java:557)
  org.apache.geode.distributed.internal.direct.DirectChannel.sendToMany(DirectChannel.java:336)
  org.apache.geode.distributed.internal.direct.DirectChannel.sendToOne(DirectChannel.java:251)
  org.apache.geode.distributed.internal.direct.DirectChannel.send(DirectChannel.java:616)
  org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.directChannelSend(GMSMembershipManager.java:1692)
  org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.send(GMSMembershipManager.java:1870)
  org.apache.geode.distributed.internal.ClusterDistributionManager.sendViaMembershipManager(ClusterDistributionManager.java:2865)
  org.apache.geode.distributed.internal.ClusterDistributionManager.sendOutgoing(ClusterDistributionManager.java:2785)
  org.apache.geode.distributed.internal.ClusterDistributionManager.sendMessage(ClusterDistributionManager.java:2824)
  org.apache.geode.distributed.internal.ClusterDistributionManager.putOutgoing(ClusterDistributionManager.java:1523)
  org.apache.geode.distributed.internal.StartupMessage.process(StartupMessage.java:280)
  org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:367)
  org.apache.geode.distributed.internal.DistributionMessage$1.run(DistributionMessage.java:432)
  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  org.apache.geode.distributed.internal.ClusterDistributionManager.runUntilShutdown(ClusterDistributionManager.java:949)


Environment

Product Version: 1.6

Resolution

This happens when locators are not able to talk to VMware GemFire servers on the VMware GemFire tcp server port 40405. This port was introduced in  VMware Tanzu GemFire 1.6.0 VMware Tanzu GemFire documentation missed mentioning that this port should be opened at the IaaS layer. Port 40405 for tcp should be opened on all VMware Tanzu GemFire service instance VMs at the IaaS layer (openstack/AWS/vSphere etc).

Powered by Wolken Software