Greenplum Error: "[CRITICAL]:-gpsegstart.py failed. (Reason='Local Software Version does not match what is expected"

Products

• VMware Tanzu Greenplum

Issue/Introduction

Symptoms:

After a Greenplum binary upgrade, a database fails to start. In the example below, this happens after upgrading from 4.3.5.4 to 4.3.13.0.

Error Message:

20170525:08:18:57:032519 gpstart:0978-grnplm-t01:gpadmin-[INFO]:-DBID:20 FAILED host:'0978-grnplm-t06' datadir:'/gpdata1/primary/gpseg18' with reason:'cmd had rc=2 completed=True halted=False
stdout='20170525:08:18:56:013977 gpsegstart.py_0978-grnplm-t06:gpadmin:0978-grnplm-t06:gpadmin-[INFO]:-Starting gpsegstart.py with args: -C en_US.utf8:en_US.utf8:en_US.utf8 -M mirrorless -V postgres (Greenplum Database) 4.3.13.0 build 1 -n 24 --era 206aa2f3f4a1ef89_170525081833 -t 600 -D 20|18|p|p|s|u|0978-grnplm-t06|sdw4|40000|None|/gpdata1/primary/gpseg18|| -D 21|19|p|p|s|u|0978-grnplm-t06|sdw4|40001|None|/gpdata1/primary/gpseg19|| -D 22|20|p|p|s|u|0978-grnplm-t06|sdw4|40002|None|/gpdata1/primary/gpseg20|| -D 23|21|p|p|s|u|0978-grnplm-t06|sdw4|40003|None|/gpdata2/primary/gpseg21|| -D 24|22|p|p|s|u|0978-grnplm-t06|sdw4|40004|None|/gpdata2/primary/gpseg22|| -D 25|23|p|p|s|u|0978-grnplm-t06|sdw4|40005|None|/gpdata2/primary/gpseg23||
20170525:08:18:56:013977 gpsegstart.py_0978-grnplm-t06:gpadmin:0978-grnplm-t06:gpadmin-[CRITICAL]:-gpsegstart.py failed. (Reason='Local Software Version does not match what is expected.
The local software version is: 'postgres (Greenplum Database) 4.3.5.4 build 1'
But we were expecting it to be: 'postgres (Greenplum Database) 4.3.13.0 build 1'
Please review and correct') exiting...
'

Environment

Cause

During upgrading Greenplum, the upgrade process failed to replace a symbolic link to the new installation. To verify this, check the current symbolic link:

[gpadmin@mdw ~]$ gpssh -f hostfile "ls -latrh /usr/local | grep greenplum-db"
[ mdw] drwxr-xr-x 13 gpadmin gpadmin 4.0K Sep 28 2016 greenplum-db-4.3.5.4
[ mdw] drwxr-xr-x 11 gpadmin gpadmin 4.0K May 25 08:01 greenplum-db-4.3.13.0
[ mdw] lrwxrwxrwx 1 root root 32 May 25 08:10 greenplum-db -> /usr/local/greenplum-db-4.3.13.0
[sdw1] lrwxrwxrwx 1 gpadmin gpadmin 20 Sep 22 2016 greenplum-db -> greenplum-db-4.3.5.4
[sdw1] drwxr-xr-x 13 gpadmin gpadmin 4.0K Sep 28 2016 greenplum-db-4.3.5.4
[sdw1] drwxr-xr-x 11 gpadmin gpadmin 4.0K May 25 08:01 greenplum-db-4.3.13.0
[sdw4] lrwxrwxrwx 1 gpadmin gpadmin 20 Sep 22 2016 greenplum-db -> greenplum-db-4.3.5.4
[sdw4] drwxr-xr-x 13 gpadmin gpadmin 4.0K Sep 28 2016 greenplum-db-4.3.5.4
[sdw4] drwxr-xr-x 11 gpadmin gpadmin 4.0K May 25 08:01 greenplum-db-4.3.13.0
[smdw] lrwxrwxrwx 1 gpadmin gpadmin 20 Sep 22 2016 greenplum-db -> greenplum-db-4.3.5.4
[smdw] drwxr-xr-x 13 gpadmin gpadmin 4.0K Sep 28 2016 greenplum-db-4.3.5.4
[smdw] drwxr-xr-x 11 gpadmin gpadmin 4.0K May 25 08:01 greenplum-db-4.3.13.0
[sdw3] lrwxrwxrwx 1 gpadmin gpadmin 20 Sep 22 2016 greenplum-db -> greenplum-db-4.3.5.4
[sdw3] drwxr-xr-x 13 gpadmin gpadmin 4.0K Sep 28 2016 greenplum-db-4.3.5.4
[sdw3] drwxr-xr-x 11 gpadmin gpadmin 4.0K May 25 08:01 greenplum-db-4.3.13.0
[sdw2] lrwxrwxrwx 1 gpadmin gpadmin 20 Sep 22 2016 greenplum-db -> greenplum-db-4.3.5.4
[sdw2] drwxr-xr-x 13 gpadmin gpadmin 4.0K Sep 28 2016 greenplum-db-4.3.5.4
[sdw2] drwxr-xr-x 11 gpadmin gpadmin 4.0K May 25 08:01 greenplum-db-4.3.13.0

Resolution

Follow the steps below to resolve this issue (Make sure to use the correct Greenplum version that has been installed):

sudo su -
source /usr/local/greenplum-db-4.3.13.0/greenplum_path.sh
gpssh -f hostfile
=> ls -altrh /usr/local/ | grep greenplum-db
=> rm -f /usr/local/greenplum-db
=> ln -s /usr/local/greenplum-db-4.3.13.0 /usr/local/greenplum-db
=> chown -R gpadmin:gpadmin /usr/local/greenplum-db
=> ls -altrh /usr/local/ | grep greenplum-db

Source a new path and you should be able to start the Database:

source /usr/local/greenplum-db/greenplum_path.sh
gpstart -a