How to rebalance your Partitioned Region buckets evenly across members in VMware GemFire
Article ID: 294439
Updated On:
Products
Issue/Introduction
When using VMware GemFire with partitioned regions (PRs), if you are ever experiencing an imbalance of data or client load, it is perhaps due to an imbalance of partitioned region buckets across your different members.
In some cases, this is possible if you have run a "rebalance" command. Using the GemFire rebalance functionality is essential when using partitioned regions. The goal is to keep the data balanced as much as possible to achieve stability in your environment.
By default, GemFire rebalances data based upon total data size in bytes across members. In general, this is completely effective to achieve enough balance to provide for stability. Stability is generally defined as a relatively even distribution across members of CPU utilization, heap consumption, and client load activity. Any of these three being out of balance can be potential sources of instability.
Symptoms can range from higher than expected CPU utilization, higher than normal Heap, and imbalance of client connections to some members versus others.
Normally, having an even distribution of data in bytes has the effect of driving a balance of partitioned region buckets. Each PR is broken down into small bucket regions, 113 primary buckets by default for a given region. See "total-num-buckets". When these buckets are evenly distributed across hosting members sufficiently, we normally see a balanced CPU, Heap, and client load activity per member.
With long lived data, the default rebalance algorithm using data size (in bytes) to determine balance is very effective.
However, when you have short lived data, such as short lived Sessions, it is possible that a rebalance run successfully - while delivering "short term" balance, it may in fact cause the bucket counts per member to be very imbalanced. Later, as sessions/entries come and go, it is possible and even likely that the members with an abnormally high number of buckets sees CPU issues, Heap issue, etc.
This is due to the fact that the some buckets at the time of the rebalance may not have had many Sessions hashed to those buckets (BUCKETSlowcount), while other buckets (BUCKETShighcount) may have had many entries. In that case, it is very possible that many of the "lowcount" buckets end up on the same member as the result of the rebalance, while the "highcount" buckets drive fewer buckets on members as the result of the rebalance.
However, it is possible that many new Sessions keys are hashed to the old "lowcount" buckets. This now means you have many buckets on Member A, and now many of those buckets could have "highcount", which can drive CPU, Heap, and client load toward that member during a burst of activity. Perhaps the old "highcount" buckets are actually receiving fewer Sessions due to the current Keys being hashed.
In any case, this default rebalance algorithm has been seen to be insufficient in such cases. Another option is needed, and it already exists in the product.
We have a "BucketCountLoadProbe," which when configured, will balance buckets not using data size, but purely by bucket counts. It achieves this behavior by giving all buckets equal weight, rather than giving them weight based upon actual bytes used in the bucket.
To use this GemFire system property, you must set the "gemfire.ResourceManager.PR_LOAD_PROBE_CLASS" property as follows:
-Dgemfire.ResourceManager.PR_LOAD_PROBE_CLASS=org.apache.geode.internal.cache.partitioned.BucketCountLoadProbe =========================
If you have been unable to achieve sufficient balance, and need better balance given the CPU and Heap resources provisioned in your environment, you may want to consider this load probe.
Note: It is very important to test this system property to make sure that your region keys get hashed evenly. If they do not, it could lead to worse balance in total. For example, if your buckets are evenly distributed across 4 members, each member having roughly 28 buckets achieves bucket balance. However, what if the show metrics command shows that your entry count differs greatly for a given region per member? This would imply that the hashing of entries to buckets may not be driving optimal balance, and another hashing algorithm or "total-num-buckets" bucket count are worth considering.
Environment
OS: ALL
Resolution
To identify whether you have balance for any region R in your environment, the following gfsh command is quite helpful:
gfsh>show metrics --region=R --member=M (you can add categories=partition if you like, but this filters the output more)
You can examine both the primary bucket count and the total bucket count on each member for each region to determine whether you have balance. If you do not, perhaps due to using our default rebalance, you can use this command to rebalance, even if for only one "Sessions" region:
gfsh>rebalance --include-region=Sessions
Using this command, combined with the above show metrics command, can give you valuable insight into your data/bucket balance. When you have such balance, you can monitor the system for even client load.
GemFire rebalance has two different behaviors:
- Driving data balance in bytes
- Driving bucket count balance
Hopefully, the bucket count balance would also deliver entry count balance, and in turn, clients accessing data would also be balanced, keeping CPU and Heap both at healthy levels.
Feedback
