How to enable Garbage Collection logging for Hadoop NameNode and DataNode service

Products

• Services Suite

Issue/Introduction

This article explains how to enable Garbage Collection logging for Hadoop NameNode and DataNode services.

Environment

Resolution

Follow the instructions below to enable Garbage Collection logging for Hadoop NameNode services. 
 

1. Open /etc/gphd/hadoop/conf/hadoop-env.sh and look for the following line:

export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote -Xms2048m -Xmx2048m -Dhadoop.security.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,RFAAUDIT -XX:ParallelGCThreads=8 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+HeapDumpOnOutOfMemoryError -XX:ErrorFile=${HADOOP_LOG_DIR}/hs_err_pid%p.log $HADOOP_NAMENODE_OPTS"

2. Insert the following into the HADOOP_NAMENODE_OPTS parameter: 

-verbose:gc

-XX:+PrintGCDetails

-Xloggc:${HADOOP_LOG_DIR}/hadoop-hdfs-namenode-`date +'%Y%m%d%H%M'`.gclog

-XX:+UseGCLogFileRotation

-XX:NumberOfGCLogFiles=20

3. After adding the GC log parameters, the HADOOP_NAMENODE_OPTS parameter should look like this: 

export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote -Xms2048m -Xmx2048m -Dhadoop.security.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,RFAAUDIT -XX:ParallelGCThreads=8 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+HeapDumpOnOutOfMemoryError -XX:ErrorFile=${HADOOP_LOG_DIR}/hs_err_pid%p.log -verbose:gc -XX:+PrintGCDetails -Xloggc:${HADOOP_LOG_DIR}/hadoop-hdfs-namenode-`date +'%Y%m%d%H%M'`.gclog -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=20 $HADOOP_NAMENODE_OPTS"

4. Restart hadoop-hdfs-namenode.

Follow the instructions below to enable Garbage Collection logging for Hadoop DataNode services.
 

1. Open /etc/gphd/hadoop/conf/hadoop-env.sh and look for the following line:

export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote -Xms2048m -Xmx2048m -Dhadoop.security.logger=ERROR,DRFAS $HADOOP_DATANODE_OPTS"

2. Insert the following into the HADOOP_DATANODE_OPTS parameter:

-verbose:gc -XX:+PrintGCDetails -Xloggc:${HADOOP_LOG_DIR}/hadoop-hdfs-datanode-`date +'%Y%m%d%H%M'`.gclog -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=20

3. After adding the GC log parameter, the HADOOP_DATANODE_OPTS parameter should look like this:

export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote -Xms2048m -Xmx2048m -Dhadoop.security.logger=ERROR,DRFAS -verbose:gc -XX:+PrintGCDetails -Xloggc:${HADOOP_LOG_DIR}/hadoop-hdfs-datanode-`date +'%Y%m%d%H%M'`.gclog -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=20 $HADOOP_DATANODE_OPTS"