GemFire: How to determine optimal total-num-buckets settings for Partitioned Regions
Article ID: 294461
Updated On:
Products
Issue/Introduction
So, the question becomes "How should we select the appropriate value for total-num-buckets for a given Partitioned Region?"
Environment
OS: ALL
Resolution
First, this article becomes irrelevant if customers are not focusing on achieving balance in the environment. This requires well designed hashcodes to achieve a great distribution of data, so that when assigning entries (<key,value> pairs) to buckets, there is a great distribution, where you see great balance of keys to buckets. In addition, making sure you are balancing buckets across the membership optimally is also needed. You can use the default data size load probe, which balances "heap" to state it simply, or the BucketCountLoadProbe which will balance BUCKETS. Hopefully, by balancing buckets you are indeed also balancing heap as a result.
There are other related KB's that discuss the above. This KB article is focused on selecting an optimal total-num-buckets setting. This requires defining what we mean by "optimal" in this selection process.
There are 2 goals.
A) For your primary bucket count (total-num-buckets setting), have no more than a 5% difference in bucket counts. The reason for this goal is to achieve the best balance of heap and, more importantly, client load. When some small subset of members are seeing a 10%+ client load, it is simply opening up the system to vulnerability, with cpu consumption issues, heap, etc.
B) Use a prime number that both satisfies the above suggestion, while also having the most members using the same number of buckets, while only 1 or 2 members use 1 fewer bucket.
For the purposes of explanation, 4 examples are included here below to assist. You can follow this formula to determine a great total-num-buckets setting for your environment.
=========
Example 1: 14 Cache Servers
The 1st step is always to multiple by number of cache servers by 20. This is due to satisfying the first suggestion of a maximum 5% disparity. 14*20=280. So, find some prime number greater than 280 that will work out nicely.
Possibilities 281, 283, 293. Trying 293.
CS = 14 => buckets(prime)= 13x + x-1 = 293 | 14x-1=293 | 14x=294 | x=294/14=21
To explain the above formula, Let x = the bucket count that will exist on all members except 1. Let CS = Number of cache servers hosting region.
293 = (CS-1)x + (1)(x-1)
Note, the (1) in parentheses above is simply designating explicitly the 1 member that will have 1 less bucket, identified as (x-1) buckets. For a 14 cache server system, this equation simplifies to:
293=13x + (x-1)
293=14x - 1
294=14x
294/14=x
So x = 21. This indicates that 13 members should have 21 buckets, and the remaining member should have 20 buckets. To confirm the math.
13members each with 21 buckets gives 273 buckets
1 member with 20 buckets gives 20 buckets.
273+20=293. Confirmed.
This satisfies both goals. Most members will have 21 buckets, sharing the bulk of the load in the most balanced fashion, with only 1 slightly less loaded member having 20 buckets. This is near your 5% disparity goal.
If you believe in your environment that a greater disparity would be fine, such as some difference up to 10% or so, that is fine. In this article, we are fine tuning, and balancing the system to the greatest degree possible such that 1 member is not overwhelmed.
=========
Example 2: 10 Cache Servers
Multiply 10 by 20 to get close to 5% disparity = 200
Nearest primes are 197,199,211. Trying 199.
CS = 10 => buckets(prime)= 9x + x-1 = 199 | 10x-1=199 | 10x=200 | x=200/10=20
Let x=number of buckets on most members
Therefore (x-1) = number of buckets on the remaining members.
199=9x + 1(x-1)
199=10x-1
200=10x
20=x
Therefore 9 members would have 20 buckets, while 1 member would have 19 buckets. (9*20) + (1*19)=180+19=199. Confirmed. Achieves both goals.
========
Example 3: 3 Cache Servers
Multiply 3 by 20 to get close to 5% disparity = 60
Perhaps just using the default=113 worth keeping, as this will achieve the best results with much less than 5% disparity (closer to 3%) That said, following the recipe gives the following:
Nearest primes are 59,61,67. Trying 59.
59=2x+ 1(x-1)
59=3x - 1
60=3x
20=x
So, 2 members each have 20 buckets, while the third member has 19 buckets.
(2*20) + (1*19) = 40 + 19 = 59 buckets. Confirmed.
The above satisfied the suggestions, but if you were to use the default 113, we have the following result:
CS = 3 => buckets(prime)= 2x + x-1 = 113 | 3x-1=113 | 3x=114 | x=114/3=38
113=2x + 1(x-1)
113=3x - 1
114=3x
38=x
So, 2 members would have 38 buckets, while the third would have 37. This achieves both goals while keeping the disparity of load below 3%.
CS = 3 => buckets(prime)= 2x + x-1 = 113 | 3x-1=113 | 3x=114 | x=114/3=38
========
Example 4: 4 Cache Servers
Multiply 4 by 20 to get close to 5% disparity = 80
Again, perhaps just using the default=113 worth keeping, as this will achieve the best results with much less than 5% disparity (closer to 3%) That said, following the recipe gives the following:
Nearest primes are 79, 83, 89. Trying 79.
79 = 3x + 1(x-1)
29 = 4x + 1
30 = 4x
7.5 = x (Not an integer, just means that it's not as optimal perhaps as another prime number... Let's try 83
83 = 3x + 1(x-1)
83 = 4x + 1
84 = 4x
21 = x
So, 3 members would have 21 buckets, while the 4th member would have 20 buckets. (3 * 21) + (1 * 20) = 63 + 20 = 83. Confirmed. This satisfies the criteria. Still, you can always test higher prime numbers if you want less than the stated goal of 5% disparity. Trying 107 prime number with 4 members:
CS = 4 => buckets(prime)= 3x + x-1 = 107 | 4x-1=107 | 4x=108 | x=108/4=27
107 = 3x + 1(x-1)
107 = 4x - 1
108 = 4x
27 = x
So, 3 members with 27 buckets, and 1 member with 26, gives (3 * 27) + (1 * 26)= 107.
The disparity would be less than a 4% disparity.
============
To summarize, the goal of this article is to help customers choose a total-num-buckets setting that makes sense and gives you optimal chance to achieve balance, while not overwhelming some small number of members.
It is important to select the correct value when first creating a system, because altering the total-num-buckets requires an export, change configuration, and import process to make such a change.
One final point is to consider the number of keys to be stored in a region. In some rare cases, sometimes we see very few entries being stored in a region. Please make sure to consider this piece of the puzzle. It would make no sense, for example, to have 113 default buckets for a region known to have at most 5 entries.
Related KB Articles:
https://knowledge.broadcom.com/external/article?articleId=294439
https://docs.vmware.com/en/VMware-Tanzu-GemFire/9.15/tgf/GUID-basic_config-data_entries_custom_classes-managing_data_entries.html?hWord=N4IghgNiBcIBZgM5wMYHsAmBTEBfIA
https://docs.vmware.com/en/VMware-Tanzu-GemFire/9.15/tgf/GUID-developing-partitioned_regions-configuring_bucket_for_pr.html
Feedback
