What Artifacts to Collect when GemFire Native Client is Not Responding

Products

• VMware Tanzu Gemfire

Issue/Introduction

The scope of this article is to provide a list of artifacts to be collected when the GemFire Native Client (NC) is not responding. Please note that some of these steps should be taken before restarting the clients and the servers.

If you experience a situation where the NC applications are in a non-responsive or a hung state, make sure to collect the following artifacts.

Symptoms:

One or more of your native client applications appear to be in a non-responding or hung state.

Environment

OS: All supported

Resolution

Checklist:

Note that the thread dumps should be taken before any restarts of the clients or the servers, and the client and server thread dumps should be taken as close in time as possible.

Client Side:

• Dump file on Windows: Before restarting any of the client applications, create a dump file on all the client machines. The easiest way to do this is to use the task manager, right click on the process and select "Create dump file." The command will return a dialog when completed along with the path to the dump file that will have a *.dmp extension. Creating dump files using proprietary monitoring software is not recommended, as we might not have the setup needed to open and read the proprietary files. 
• Statistics and log files: Logs and statistics files (*.gfs) from each of the clients.
• Configuration files: The cache.xml and the property files for the client side.

Server Side:

• Thread Dumps: In case you are restarting the servers, please make sure to take the thread dump on all the servers before restarting. Information on how to take thread dumps can be found here.
• Statistics and Log files: Collect the statistics files and the log files from each of the members (including locators) of the distributed system. Make sure the logs and statistics files that you sent, cover the time of the incident.
• GC Logs: Collect the available GC logs.
• It would be good if you change the log level to "finer" for about 5 minutes and change it back to what you had earlier right after, and then send the log files to the servers. Please make sure that you revert back this change. If not, your disk will get full with logs. Log level can be changed using the GFSH alter runtime command.
• Show Deadlock: Collect the output of the GFSH show deadlock command.