GPText index is corrupted in both primary and mirror

Products

• VMware Tanzu Greenplum

Issue/Introduction

Symptoms:

GPText instances are failing to recover due to a corrupt index.

Error message

gptext-recover
20160222:09:04:17:010108 gptext-recover:mdw1:gpadmin-[INFO]:-Retrieving configuration from database...
20160222:09:05:22:010108 gptext-recover:mdw1:gpadmin-[INFO]:-List of down GPText instances:
20160222:09:05:22:010108 gptext-recover:mdw1:gpadmin-[INFO]:-Content id: 0
20160222:09:05:22:010108 gptext-recover:mdw1:gpadmin-[INFO]:- Role: Primary
20160222:09:05:22:010108 gptext-recover:mdw1:gpadmin-[INFO]:- Host: sdw1.mgmt
20160222:09:05:22:010108 gptext-recover:mdw1:gpadmin-[INFO]:- Seg Directory: /data1/primary/gpseg0
20160222:09:05:22:010108 gptext-recover:mdw1:gpadmin-[INFO]:-Content id: 1
20160222:09:05:22:010108 gptext-recover:mdw1:gpadmin-[INFO]:- Role: Primary
20160222:09:05:22:010108 gptext-recover:mdw1:gpadmin-[INFO]:- Host: sdw1.mgmt
20160222:09:05:22:010108 gptext-recover:mdw1:gpadmin-[INFO]:- Seg Directory: /data2/primary/gpseg1
20160222:09:05:22:010108 gptext-recover:mdw1:gpadmin-[INFO]:-Content id: 10
20160222:09:05:22:010108 gptext-recover:mdw1:gpadmin-[INFO]:- Role: Primary
20160222:09:05:22:010108 gptext-recover:mdw1:gpadmin-[INFO]:- Host: sdw3.mgmt
20160222:09:05:22:010108 gptext-recover:mdw1:gpadmin-[INFO]:- Seg Directory: /data1/primary/gpseg10
20160222:09:05:22:010108 gptext-recover:mdw1:gpadmin-[INFO]:-merit.dbo.contents_mfd_1_prt_p20150615 index is corrupted in both primary and mirror. Cannot recover.
Traceback (most recent call last):
 File "/usr/local/greenplum-text-1.3.0.2/bin/gptext-recover", line 83, in <module>
 ret = recover(options,conn,array,pool)
 File "/usr/local/greenplum-text-1.3.0.2/lib/python/gptextlib/recover.py", line 359, in recover
 down_content_role_map[content_id][0][1]=remove_corrupted_index(index_lst1,corrupted_index_lst)
TypeError: 'tuple' object does not support item assignment

Environment

Cause

The index is corrupted at both the primary and mirror segments.

Resolution

To fix this issue, the index for the affected table needs to be dropped and recreated. Once this is done, run recovery again to recover the GPText instance.

A force recovery can be attempted by using the following command: 

gptext-recover -f -c <contentid> -r <role of the solr instance to be recovered>