How to change heap size of Pivotal Hadoop cluster components

Products

• Services Suite

Issue/Introduction

Most Hadoop components run as Linux processes. These processes create, manage and monitor jobs running in Hadoop cluster. These processes may run out of memory if the default heap size is not sufficient enough to handle the heavy workloads.

Environment

Resolution

Please refer to the following table of how to change the default heap size of these components.
 

Note: You need to restart the corresponding service after changing the heap size of a Hadoop component.
 

Component	Configuration file	Changing Heap size(java Xmx) example	Verification
HDFS NameNode	/etc/gphd/hadoop /conf/hadooenv.sh	export HADOOP_HEAPSIZE=2048	ps ef|grep namenode
HDFS DataNode	/etc/gphd/hadoop/conf/hadoop-env.sh	export HADOOP_HEAPSIZE=2048	ps ef|grep datanode
Yarn Resource Manger	/etc/gphd/hadoop/conf/yarn-env.sh	export YARN_RESOURCEMANAGER_HEAPSIZE=1024	ps ef|grep resourcemanager
Yarn Node Manager	/etc/gphd/hadoop/conf/yarn-env.sh	export YARN_NODEMANAGER_HEAPSIZE=1024	ps ef|grep nodemanager
Hive Server ( Hive Server2)	/etc/gphd/hive/conf/hive-env.sh	export HADOOP_CLIENT_OPS="${HADOOP_CLIENT_OPS} -Xmx2048"	ps -ef|grep HiveServer
Hive Meta Store	/etc/gphd/hive/conf/hive-env.sh	export HADOOP_CLIENT_OPS="${HADOOP_CLIENT_OPS} -Xmx2048"	ps -ef|grep HiveMetaStore
HBase Master Server	/etc/gphd/hbase/conf/hbase-env.sh	export HBASE_HEAPSIZE=1000	ps -ef |grep hbase-master
HBase Region Server	/etc/gphd/hbase/conf/hbase-env.sh	export HBASE_HEAPSIZE=1000	ps ef|grep hbase-regionserver
Zookeeper Server	/etc/gphd/zookeeper/conf/java.env	export JVMFLAGS="-Xmx2048m"	ps -ef|grep zookeeper
Secondary NameNode	/etc/gphd/hadoop/conf/hadoop-env.sh	export HADOOP_HEAPSIZE=1000	ps ef |grep secondarynamenode
mapreduce-historyserver	/etc/gphd/hadoop/conf/mapred-env.sh	export HADOOP_JOB_HISTORYSERVER_HEAPSIZE=1000	ps ef |grep historyserver

Note: Some of the Hadoop services use the same parameter in the same configuration files to set the heap size. If two daemons are collocated on the same host, changing the parameter will affect both services.

Consider Hive server2 and Hive meta store as an example. Both services uses the parameter export HADOOP_CLIENT_OPS in file /etc/gphd/hive/conf/hive-env.sh. If you just want to change Hive server2 heap size, follow the workaround below:
 

1. kdir -p /etc/gphd/hive2/conf
2. cp -rp /etc/gphd/hive/conf/* /etc/gphd/hive2/conf/
3. vi /etc/gphd/hive2/conf/hive-env.sh export HADOOP_CLIENT_OPS="${HADOOP_CLIENT_OPS} <Your_Heap_Size>"
4. vi /etc/default/hive-server2 export HIVE_CONF_DIR=/etc/gphd/hive2/conf
5. service hive-server2 restart