GPText: Reloading an Index gives unable to reload core error

Products

• VMware Tanzu Greenplum

Issue/Introduction

Reloading a GPText index using the following command will fail with the following:

select * from gptext.reload_index('demo.twitter.message') ;

org.apache.solr.common.SolrException: Unable to reload core [demo.twitter.message214_replica_n858]

Running the following query may also give this message:

select count(replica_name) as replica_count, shard_name,  regexp_replace(node_name, ':.*', '', 'g')  as hostname from gptext.index_summary('merit.dbo.products') group by (shard_name,hostname) order by 2;
Function gptext.__core_statistics_impl(anytable,text[].text.gptext.__index_context):":Failed to get core stats.Function: init, File: CoreStatisticsUDF.cpp(74)(UDF.cpp:215)(UDF.cpp:215)(seg0 slice2 172.28.8.1:1025 pid=527584) (cbdisp.c:254) CONTEXT: PL/pdSQL function "core_statistics" line 8 at FOR over SELECT rows SQL function "core_statistics" statement 1
SQL function "index_summary" statement 1

Creating an index will have similar challenges and may error out:

select * from utils.create_gptext_indexes();

WARNING: Could not find collection: demo.twitter.messages_intercept_filers. (AdminUDF.cpp:979)

create_gptext_indexes
-------------------------------
FAILURE

Solr Log will mention:

Error loading class 'com.emc.solr.handler.component.OffsetsComponent'

Environment

Product Version: 6.21

Resolution

During a GPText upgrade, a file is created /tmp/.gptext_updowngrade_file to record which nodes were restarted and which nodes weren’t. With this file, GPText could continue with the unrestarted nodes if gptext-upgrade failed instead of restarting every node from scratch.

This file should have been deleted if gptext-upgrade finished. But for some reason, the gptext-upgrade failed that time and GPText was manually recovered and upgraded. But this file wasn’t deleted.
If GPText is upgraded again, then only certain nodes will be restarted which means any new code changes will not be picked up by the unrestarted nodes.

Resolution is to remove /tmp/.gptext_updowngrade_file and restart GPText cluster.