How to set heap size of hive clients different from other hive services

Products

• Services Suite

Environment

Cause

The Heap size for all the Hive services (metastore, hiveserver and beeline clients) is the same, as per the default configuration. This may cause severe performance issues in the cluster. A user should reduce the heap sizes for all the Hive clients and thereby allow more concurrent connections to the cluster.

Resolution

Let's consider a scenario of a cluster having hive heap size of 12 GB. Then all the individual connected hive clients will have 12 GB allocated memory and would limit the number of client connections and that would result into the cluster to run out of memory.

Here is the procedure to set heap size of Hive clients different from metastore and Hive server.

Add the following changes to hive-env.sh template in Ambari and restart hive services. You will see the heap size for all the hive clients got reduced to 1 GB and thereby can have more concurrent Hive client sessions to the cluster.

if [ "$SERVICE" = "cli" ]; then

if [ -z "$DEBUG" ]; then
export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+UseNUMA -XX:+UseParallelGC -XX:-UseGCOverheadLimit"
else
export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:-UseGCOverheadLimit"
fi
fi

if [ "$SERVICE" = "metastore" ] || [ "$SERVICE" = "hiveserver2" ]; then
export HADOOP_HEAPSIZE={{hive_heapsize}} # Setting for HiveMetastore & HiveServer2
else
export HADOOP_HEAPSIZE=1024 #Client
fi

export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Xmx${HADOOP_HEAPSIZE}m"