gprecoverseg fails with "Exception: Local Software Version does not match what is expected" in Greenplum

Products

• VMware Tanzu Greenplum

Issue/Introduction

In Greenplum, gprecoverseg fails with:
 

sdw2:gpadmin-[ERROR]:-gpsegstart.py failed.  exiting...
Traceback (most recent call last):
  File "/usr/local/greenplum-db/lib/python/gppylib/mainUtils.py", line 270, in simple_main_locked
    commandObject = createCommandFn(options, args)
  File "/usr/local/greenplum-db/./sbin/gpsegstart.py", line 614, in createProgram
    logfileDirectory=logfileDirectory)
  File "/usr/local/greenplum-db/./sbin/gpsegstart.py", line 147, in __init__
    "Please review and correct" % (actual_gpversion, expected_gpversion))
Exception: Local Software Version does not match what is expected.
The local software version is: 'postgres (Greenplum Database) 5.21.5 build commit:ac49b4df3ba9fb4db39b0e45b075c66e935d798a'
But we were expecting it to be: 'postgres (Greenplum Database) 5.23.0 build commit:5eaaa5800e9f492683c3ce313e54d3db5afbce79'
Please review and correct

In this situation, one of the host was powered down for a long time and skipped the Greenplum code upgrade. You can confirm this by checking the version installed:

[gpadmin@mdw ~]$ gpssh -f hostfile "ll /usr/local | grep greenplum-db | grep '>'  "
[sdw2] lrwxrwxrwx. 1 gpadmin gpadmin 18 Oct 24 16:25 greenplum-db -> /usr/local/greenplum-db-5.17.0
[mdw]  lrwxrwxrwx. 1 gpadmin gpadmin 30 Jun 24 11:16 greenplum-db -> /usr/local/greenplum-db-5.21.0
[smdw] lrwxrwxrwx. 1 gpadmin gpadmin 30 Jun 24 11:16 greenplum-db -> /usr/local/greenplum-db-5.21.0
[sdw1] lrwxrwxrwx. 1 gpadmin gpadmin 30 Jun 24 11:16 greenplum-db -> /usr/local/greenplum-db-5.21.0

Environment

Product Version: 5.23

Resolution

In order to resolve this issue, you need to manually upgrade Greenplum. 

1. Manually copy over the /usr/local/greenplum-db-5.21.0 directory from any host to a host that reports the error.

2. Once it's there, remove the current symbolic link and create a new one pointing to the right location:

ssh sdw2 
rm -rf /usr/local/greenplum-db
ln -s /usr/local/greenplum-db-5.21.0 /usr/local/greenplum-db 
chown -R gpadmin /usr/local/greenplum-db

3. When you have this finished, check the installation version with gpssh:

gpssh -f hostfile "ll /usr/local | grep greenplum-db | grep '>' "

4. When all host point to the correct installation, try the recovery command again.