"Allocating larger network read buffer" high alert notification in GemFIre v9.10.9 or v9.10.10
Article ID: 294449
Updated On:
Products
VMware Tanzu Gemfire
Issue/Introduction
A product issue exists in GemFire v9.10.9 and v9.10.10 that drives message loss and consequently, distributed deadlocks.
Customers are not vulnerable if all members, including locators, have the same socket-buffer-size configuration. However, some customers choose to set the locator socket-buffer-size to a lower value, and this creates the potential for message loss.
If you do have different socket-buffer-size configurations across the distributed system and you see the following log message in your GemFire 9.10 logs, you are vulnerable to message loss caused by overwritten messages stored in the buffers:
Customers are not vulnerable if all members, including locators, have the same socket-buffer-size configuration. However, some customers choose to set the locator socket-buffer-size to a lower value, and this creates the potential for message loss.
If you do have different socket-buffer-size configurations across the distributed system and you see the following log message in your GemFire 9.10 logs, you are vulnerable to message loss caused by overwritten messages stored in the buffers:
[info 2021/10/13 17:14:56.177 EDT locator <P2P message reader for x.y.1.5(server1:8438)<v1>:41001 shared unordered sender uid=1 local port=42181 remote port=57345> tid=0x4c] Allocating larger network read buffer, new size is 134074 old size was 32768.
Environment
Product Version: 9.10
OS: ALL
OS: ALL
Resolution
To eliminate any potential for subsequent loss due to overwritten messages, update all members to the same socket-buffer-size configurations. The safest path would be to select the larger size, and set all members to this same size.
This process will involve shutting down a member, reconfiguring the socket-buffer-size, and restarting that member, one at a time, until all members socket-buffer-sizes are equivalent.
Note: It is important that the OS level configuration is also sufficiently sized to avoid these issues. If the OS configuration is less than the final socket-buffer-size on a member, such that it remains necessary to grow and shrink these buffers, you remain vulnerable.
Confirm that your GemFire configuration for socket-buffer-size is less than or equivalent to your OS configuration limitations for socket-buffer-sizes. The following message is seen in your GemFire logs if the socket buffer size has been reduced due to OS configuration limitations:
This process will involve shutting down a member, reconfiguring the socket-buffer-size, and restarting that member, one at a time, until all members socket-buffer-sizes are equivalent.
Note: It is important that the OS level configuration is also sufficiently sized to avoid these issues. If the OS configuration is less than the final socket-buffer-size on a member, such that it remains necessary to grow and shrink these buffers, you remain vulnerable.
Confirm that your GemFire configuration for socket-buffer-size is less than or equivalent to your OS configuration limitations for socket-buffer-sizes. The following message is seen in your GemFire logs if the socket buffer size has been reduced due to OS configuration limitations:
[info 2021/10/13 17:14:56.177 EDT server1 <P2P message reader@75099de0> tid=0x4c] Socket send buffer size is 6710884 instead of the requested 16777215.
Feedback
Yes
No
Powered by
