Different behavior of data balance when VMware Tanzu GemFire Cache servers are stopped/started using Monit and BOSH

Products

• VMware Tanzu Gemfire

Issue/Introduction

This article will discuss the difference between Monit start/stop and BOSH start/stop as it relates to VMware Tanzu GemFire rebalance operations.

Stopping/Starting the individual servers vwith BOSH and Monit results in differences in the behavior when we want to make sure that the data is balanced across all the Cache Servers.

Environment

Product Version: 1.10

Resolution

The behavior of data rebalance across all Tanzu GemFire Cache servers are different when the servers are started/stopped using Monit and BOSH.

BOSH

When we stop/start the servers using BOSH, rebalance is performed automatically because BOSH can coordinate across multiple VMs. 

Monit

When we stop/start using Monit, rebalance needs to be performed explicitly because Monit only knows about its local processes. 

To ensure that the data is balanced across all the cache servers, we need to do the following:

While stopping/starting the individual servers with Monit, "rebalance" needs to be performed explicitly. Below are the steps to perform rebalance:

1. Login to gfsh using a "gfsh login string" that you can get from the service key (cf service-key <service> <key>).

2. Then perform the "rebalance". The following link shows the details on the rebalance command parameters: https://gemfire.docs.pivotal.io/910/geode/tools_modules/gfsh/command-pages/rebalance.html

3. Optionally, you can verify if the data is balanced using the following command:

show metrics --member=<member> --region=<region>

 
Then look for the following metrics within the region category in the output:

• totalRegionEntryCount 
• totalBucketCount  
• totalPrimaryBucketCount 

Additional Information

https://community.pivotal.io/s/article/GemFire-Rebalance---Achieving-and-Maintaining-Balanced-PR-Data