Standard Output and thread dumps on GemFire 9

Products

• VMware Tanzu Gemfire

Issue/Introduction

Symptoms:

When issuing a KILL -3 to the GemFire java process, after starting GemFire member using the gfsh start command, the thread dump is not showing in the member's log file. It also does not appear in the console output where the command was executed.

Any output coming from the JVM itself, or any System.out.println output added for debugging in custom code, is lost in GemFire 9 when the member is started using GFSH.

Environment

Cause

This didn't happen with earlier versions of GemFire when the member was started using the already deprecated start scripts found in the product /bind/directory. In that scenario there was always a start_cacheserver.log file that would capture any output that was outside of the logging framework.

Disabling output redirection by default was done on purpose to address several requests about the fact that GemFire was hi-jacking STDOUT and STDERR. For example, when you run GemFire within Tomcat, all logging and STDOUT and STDERR were being redirected to the GemFire log file as soon as the Cache was created.

Resolution

Getting thread dumps

When using gfsh to start the server (or locator) you can execute the following command to get a stack trace:

export stack-traces --file=[file name] --member=[member name]

Getting Standard Output in the logs

This can be achieved by using two hidden parameters:

--J=-Dgemfire.OSProcess.DISABLE_OUTPUT_REDIRECTION=false
--J=-Dgemfire.OSProcess.ENABLE_OUTPUT_REDIRECTION=true

Both have to be set but will later on be replaced with a single flag in the case of using GFSH. This configuration still doesn't allow the KILL -3 to get though, so you would need to use the gfsh export  command as previously explained.

Solution in GemFire 9.5 and later

In GemFire 9.5 the --redirect-output flag was added, which can be used at GFSH startup and will allow the user to control the STDOUT and STDERR behavior.