Running gprecoverseg to recover failed segments errors out with misleading message due to incorrect GPHOME setting across segments.
gprecoverseg failed. (Reason='Inconsistency in catalog and segment Role/Mode. Catalog Role = p. Segment Mode = line 7.')
When executed with verbose mode on (-v) it shows error message in stderr:
20150827:06:27:23:008558 gprecoverseg:mdw:gpadmin-[DEBUG]:-[worker15] finished cmd: Get segment status information cmdStr='ssh -o 'StrictHostKeyChecking no' sdw2 ". /usr/local/greenplum-db/./greenplum_path.sh; $GPHOME/bin/gp_primarymirror -h sdw2 -p 46002"' had result: cmd had rc=1 completed=True halted=False stdout='' stderr='/home/gpadmin/.bashrc: line 7: /usr/local/test/greenplum_path.sh: No such file or directory mode: PrimarySegment segmentState: Ready dataState: InChangeTracking faultType: NotInitialized mode: PrimarySegment segmentState: Ready dataState: InChangeTracking faultType: NotInitialized
The issue is due to environment variable "GPHOME", used by gprecoverseg python script, not being set in .bashrc file across segments.
Edit file .bashrc across segments to export environment variable GPHOME and retry the gprecoverseg command again.
export GPHOME=/usr/local/greenplum-db