How to clear sender queues
Article ID: 294032
Updated On:
Products
Issue/Introduction
A Tanzu GemFire cluster uses a gateway sender to distribute region events to another, remote Tanzu GemFire cluster. You can create multiple gateway sender configurations to distribute region events to multiple remote clusters, and/or to distribute region events concurrently to another remote cluster. A gateway sender always communicates with a gateway receiver in a remote cluster.
An asynchronous event listener generalizes the operation of the gateway sender to allow user-defined behaviors. The asynchronous event queue is then an ordered collection of events that occurred in a region, e.g., creating a new entry, or updating an existing entry. An asynchronous event listener has its method invoked from time to time with batches of events that have occurred previously.
Tanzu GemFire uses gateway sender queues to distribute events for regions that are configured with a gateway sender. AsyncEventListeners also use an asynchronous event queue to distribute events for configured regions. You can configure a gateway sender queue or an asynchronous event queue to persist data to disk similar to the way in which replicated regions are persisted.
There are a number of scenarios where such queues may need to be manually cleared. For example:
- The gateway sender is in some corrupted state;
- The event queues are not draining due to external environmental-related issues (i.e. network disruptions, load balancer or firewall issues, or other connectivity issues);
- The persistent disk store for the gateway sender or async event queue had become corrupt or missing;
- Poor performance with queue backups; or
- The cluster-related issues such as insufficient heap, disk space, or CPU.
This article provides guidance on how to clear event queues in these situations.
Resolution
Below is an example of how to clear a gateway sender queue, but the process for asynchronous event queues is analogous. Please refer to the References below, for more detailed information.
NOTE: This process assumes that the gateway / async event queue has its own disk store separate from any regions, other queues, or the PdxType registry. If this is not the case STOP, you cannot safely clear the queue using this method.
Procedure:
1. Stop the Gateway Receivers using the gfsh command "stop gateway-receiver".
2. Confirm that no Gateway Receivers are running using the gfsh command “status gateway-receiver”.
3. Remove/clean up the disk stores associated with the gateway sender on all the nodes, making sure to take a backup of the disk stores if the disk store data may ever be needed. To do this you will need to:
- Identify the location of the disk store by reviewing your config (or possibly logs). The path will be relative to the cache server’s working directory if not given as an absolute path.
- Remove (or move aside) any files associated with the disk store. Assuming the disk store name is “abc”, the associated files will be named OVERFLOWabc*, BACKUPabc*, or DRLK_IFabc*. Any other disk store files (e.g. "BACKUPdefault*") should be left alone.
4. If there are heap issues when bringing up any nodes, temporarily increase the available memory (an additional 25% of memory should be sufficient to bring the nodes up).
5. Bring up the nodes with gateway senders last,
6. Start the gateway senders by using the gfsh command “start gateway-sender”
7. Start the gateway receivers by using the gfsh command “start gateway-receiver”
References:
Overview of Multi-site Caching
Configuring a Multi-site (WAN) System
Configuring Multi-Site (WAN) Event Queues
Implementing an AsyncEventListener for Write-Behind Cache Event Handling
stop gateway-receiver
status gateway-receiver
start gateway-receiver
start gateway-sender
Feedback
