VMware Gemfire Logging mechanisms behaves differently when migrating servers to recent releases

Products

• VMware Tanzu Gemfire

Issue/Introduction

Some VMware Gemfire users were unable to find server logs after upgrading to a recent Gemfire release, e.g. v9.9.x, v9.10.x, with custom log4j.xml, which can be reproduced as following (using VMware Gemfire v9.9.1 as the example). The gfsh output says that the server log has been generated but it is not present in the directory.

Note: the same gfsh script could be used for older Gemfire releases without issues.

gfsh>start server --name=server1 --server-port=40411 --J=-Dlog4j.configurationFile=/path/to/gemfire/config/log4j2.xml
Starting a Geode Server in /gemfire/pivotal-gemfire-9.9.1/server1...
Server in /gemfire/pivotal-gemfire-9.9.1/server1 on mbp.lan[40411] as server1 is currently online.
Process ID: 97062
Uptime: 2 seconds
Geode Version: 9.9.1
Java Version: 1.8.0_66
Log File: /gemfire/pivotal-gemfire-9.9.1/server1/server1.log
JVM Arguments: -Dgemfire.default.locators=192.168.86.21[10334] -Dgemfire.start-dev-rest-api=false -Dgemfire.use-cluster-configuration=true -Dlog4j.configurationFile=/path/to/gemfire/config/log4j2.xml -XX:OnOutOfMemoryError=kill -KILL %p -Dgemfire.launcher.registerSignalHandlers=true -Djava.awt.headless=true -Dsun.rmi.dgc.server.gcInterval=9223372036854775806
Class-Path: /gemfire/pivotal-gemfire-9.9.1/lib/geode-core-9.9.1.jar:/gemfire/pivotal-gemfire-9.9.1/lib/geode-dependencies.jar

user$ cat /gemfire/pivotal-gemfire-9.9.1/server1/server1.log
cat: /gemfire/pivotal-gemfire-9.9.1/server1/server1.log: No such file or directory

Environment

Product Version: 9.9

Resolution

In general, VMware Gemfire users need to selectively copy from the default log4j2.xml within the downloaded release to their custom log4j xml file in order to get things to work properly according to the Gemfire document. The main reason, the recent releases (e.g. v9.9.x, v9.10.x) might behave differently is that Geode no longer dynamically adds/removes custom appenders — instead Geode now requires that GeodeConsole, GeodeLogWriter, GeodeAlert be specified in the log4j2.xml and then Geode tells those appenders to turn on and off without removing them from log4j2.

To fix this issue, check out the following steps to make sure GeodeConsole, GeodeLogWriter, GeodeAlert are added to the custom log4j2 xml file:

1. Check whether the geode packages has been added to the Configuration line and add it if not:

<Configuration shutdownHook="disable" packages="org.apache.geode">

2. Check whether the geode-pattern property has been added to Properties tag and add it if not:

<Property name="geode-pattern">[%level{lowerCase=true} %date{yyyy/MM/dd HH:mm:ss.SSS z} &lt;%thread&gt; tid=%hexTid] %message%n%throwable%n</Property>

3. Check whether geode LogWriter and Alert appenders have been added as appenders and add these appenders if not:

<Appenders>
    <GeodeConsole name="STDOUT" target="SYSTEM_OUT">
      <PatternLayout pattern="${geode-pattern}"/>
    </GeodeConsole>
    <GeodeLogWriter name="LOGWRITER">
      <PatternLayout pattern="${geode-pattern}"/>
    </GeodeLogWriter>
    <GeodeLogWriter name="SECURITYLOGWRITER" security="true">
      <PatternLayout pattern="${geode-pattern}"/>
    </GeodeLogWriter>
    <GeodeAlert name="ALERT"/>
  </Appenders>

4. Check whether AppenderRefs for Geode appenders have been added to Loggers and add them if not:

  <Loggers>
    <Logger name="com.gemstone" level="INFO" additivity="true"/>
    <Logger name="org.apache.geode" level="INFO" additivity="true">
    <filters>
      <MarkerFilter marker="GEODE_VERBOSE" onMatch="DENY" onMismatch="NEUTRAL"/>
    </filters>
    </Logger>
    <Logger name="org.apache.geode.security" level="INFO" additivity="false">
      <AppenderRef ref="SECURITYLOGWRITER"/>
    </Logger>
    <Logger name="org.jgroups" level="FATAL" additivity="true"/>
    <Logger name="org.eclipse.jetty" level="FATAL" additivity="true"/>
    <Root level="INFO">
      <AppenderRef ref="STDOUT"/>
      <AppenderRef ref="LOGWRITER"/>
      <AppenderRef ref="ALERT"/>
    </Root>
  </Loggers>

Note: These changes should be necessary for both locators and servers.