How to set Java Heap Space for GPText Solr instances
search cancel

How to set Java Heap Space for GPText Solr instances

book

Article ID: 296171

calendar_today

Updated On:

Products

VMware Tanzu Greenplum

Issue/Introduction

Symptoms:
To avoid out of memory (OOM) errors, It is necessary to increase the Java Heap Space.

Environment


Cause

Sometimes it is necessary to increase the Java Heap Space to avoid out of memory (OOM) errors. An example of an OOM error is below:

SQL_ERROR. Class: 'XX', Condition: '000', Detail: 'ERROR: Error sending data. java.lang.OutOfMemoryError: Java heap space (solrSay goodbye to copy and paste! Use quick text for common notes, phrases, and messages. Quick text is supported in email, chat, notes, Knowledge, and more!_curl.c:826) (seg64 slice5 sdw11:40004 pid=10240) (cdbdisp.c:1326); Error while executing the query'

Note: It could be the case that the Java instances are down and there is no obvious information about the failure in the log.

If this is the case, the issue is caused by the JAVA Heap Space size. While starting up the gptext processes, it will show the following: 

gpadmin   17175      1  0 07:30 ?        00:00:39 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64/jre/bin/java -server -Xms1024M -Xmx2048M -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/data/gptext/primary2/solr0/logs/solr_gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=9 -XX:GCLogFileSize=20M -DzkClientTimeout=15000 -DzkHost=gptext-00:2188,gptext-00:2189,gptext-00:2190/gptext -Dsolr.log.dir=/data/gptext/primary2/solr0/logs -Djetty.port=18983 -DSTOP.PORT=17983 -DSTOP.KEY=solrrocks -Dhost=gptext-00 -Duser.timezone=UTC -Djetty.home=/opt/greenplum-solr/server -Dsolr.solr.home=/data/gptext/primary2/solr0/data -Dsolr.data.home= -Dsolr.install.dir=/opt/greenplum-solr -Dsolr.default.confdir=/opt/greenplum-solr/server/solr/configsets/_default/conf -Dlog4j.configuration=file:/data/gptext/primary2/solr0/log4j.properties -Djute.maxbuffer=20000000 -Xss256k -Dsolr.jetty.https.port=18983 -Dsolr.log.muteconsole -XX:OnOutOfMemoryError=/opt/greenplum-solr/bin/oom_solr.sh 18983 /data/gptext/primary2/solr0/logs -jar start.jar --module=http

In the oom_solr.sh, it shows the following: 

SOLR_PORT=$1
SOLR_LOGS_DIR=$2
SOLR_PID=`ps auxww | grep start.jar | grep $SOLR_PORT | grep -v grep | awk '{print $2}' | sort -r`
if [ -z "$SOLR_PID" ]; then
 echo "Couldn't find Solr process running on port $SOLR_PORT!"
 exit
fi
NOW=$(date +"%F_%H_%M_%S")
(
echo "Running OOM killer script for process $SOLR_PID for Solr on port $SOLR_PORT"
kill -9 $SOLR_PID
echo "Killed process $SOLR_PID"
) | tee $SOLR_LOGS_DIR/solr_oom_killer-$SOLR_PORT-$NOW.log

Resolution

Note: The $SEGMENT_DATA_DIRECTORY here represents one of the segment data directories.
 

1. Check what is currently running and configure Java Heap Space using the following commands:

ps -ef | grep solr
gpadmin   17175      1  0 07:30 ?        00:00:39 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64/jre/bin/java -server -Xms1024M -Xmx2048M -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/data/gptext/primary2/solr0/logs/solr_gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=9 -XX:GCLogFileSize=20M -DzkClientTimeout=15000 -DzkHost=gptext-00:2188,gptext-00:2189,gptext-00:2190/gptext -Dsolr.log.dir=/data/gptext/primary2/solr0/logs -Djetty.port=18983 -DSTOP.PORT=17983 -DSTOP.KEY=solrrocks -Dhost=gptext-00 -Duser.timezone=UTC -Djetty.home=/opt/greenplum-solr/server -Dsolr.solr.home=/data/gptext/primary2/solr0/data -Dsolr.data.home= -Dsolr.install.dir=/opt/greenplum-solr -Dsolr.default.confdir=/opt/greenplum-solr/server/solr/configsets/_default/conf -Dlog4j.configuration=file:/data/gptext/primary2/solr0/log4j.properties -Djute.maxbuffer=20000000 -Xss256k -Dsolr.jetty.https.port=18983 -Dsolr.log.muteconsole -XX:OnOutOfMemoryError=/opt/greenplum-solr/bin/oom_solr.sh 18983 /data/gptext/primary2/solr0/logs -jar start.jar --module=http

$ cat <Solr Instances Dir>/solr.in.sh| grep JAVA
SOLR_JAVA_MEM="-Xms1024M -Xmx2048M"

2. Using the following command to change the Java Heap Space:

$ gptext-config jvm -o "-Xms2048M -Xmx‌10240M"

3. Confirm that the change has been made in the solr.in.sh in every Solr directory.

$ cat <Solr Instances Dir>/solr.in.sh| grep JAVA
SOLR_JAVA_MEM="-Xms2048M -Xmx10240M"

4. Restart the GPText.

$ gptext-stop
$ gptext-start

5. Check and confirm the Java Heap Space of the running Solr instances. The process should reflect the changed settings.

$ ps -ef | grep solr
gpadmin 7893 1 0 17:52 ? 00:00:05 /usr/local/greenplum-db/ext/jre-1.6.0_32/jre1.6.0_32/bin/java -server -Xms2048M -Xmx10240M -Dlog4j.configuration=file:/gpdata/segments/gpseg0/solr/lib/log4j.properties -Djetty.home=/usr/local/greenplum-text-1.3.0.3/greenplum-solr -Dsolr.solr.home=/gpdata/segments/gpseg0/solr -Djetty.port=39900 -jar /usr/local/greenplum-text-1.3.0.3/greenplum-solr/start.jar:


Powered by Wolken Software