gp_upgrade failed at initialize stage due to some segment has orphan function using old GPCC lib file

Products

• VMware Tanzu Greenplum

Issue/Introduction

Problem Summary:

-- the gp_upgrade failed at initialize stage. 
-- the error is like below: 

20220808:10:53:57 gpupgrade_cli:gpadmin:dopss01.titan.com:068291-[INFO]:-/home/gpadmin/greenplum-db-6.21.0/bin/pg_upgrade --retain --old-bindir /home/gpadmin/greenplum-db-5.29.7/bin --new-bindir /home/gpadmin/greenplum-db-6.21.0/bin --old-datadir /gpdata/master/gpseg-1 --new-datadir /gpdata/master/gpseg.nUWh1QdUpwI.-1 --old-port 5431 --new-port 50432 --mode dispatcher --old-gp-dbid 1 --new-gp-dbid 1 --check --old-options -x 50
20220808:10:54:04 gpupgrade_cli:gpadmin:dopss01.titan.com:068291-[ERROR]:-initialize: substep "CHECK_UPGRADE": 2 errors occurred:
    * check primary segment on host datepoc03.titan.com: rpc error: code = Unknown desc = check primaries: check primary on host datepoc03.titan.com with content 16: exit status 1
    * check primary segment on host datepoc02.titan.com: rpc error: code = Unknown desc = check primaries: check primary on host datepoc02.titan.com with content 10: exit status 1

-- in the above example, we can see the CHECK_UPGRADE step failed on 2 segments. 
-- if we look into the segment's gp_upgrade log, we can see the CHECK_UPGRADE called "pg_upgrade --check" command to do pre-check for upgrade
-- the "pg_upgrade --check" failed due to source cluster has some functions using the GPCC lib file which the target cluster does not have

Checking for hash and GIN indexes                           ok 80.122ms
Checking for bpchar_pattern_ops indexes                     ok 77.880ms
Checking for presence of required libraries                 fatal
| Your installation references loadable libraries that are missing from the
| new installation.  You can add these libraries to the new installation,
| or remove the functions using them from the old installation.  A list of
| problem libraries is in the file:
|     loadable_libraries.txt
 
$ cat loadable_libraries.txt
Could not load library "/home/gpadmin/gpcc-installpath/greenplum-cc-web-4.9.0/lib/gp_browser"
ERROR:  could not load library "/home/gpadmin/gpcc-installpath/greenplum-cc-web-4.9.0/lib/gp_browser.so": /home/gpadmin/gpcc-installpath/greenplum-cc-web-4.9.0/lib/gp_browser.so: undefined symbol: SnapshotNowData (dfmgr.c:240)

Environment

Product Version: 5.29

Resolution

The gp_browser.so was used by GPCC for collecting the table size. we need to find out all the functions on each database to see which function still using the GPCC lib file and drop all of them. 

For example:

# SELECT proname,probin  from pg_proc where proname = 'gpcc_filenode_scan';
      proname       |                    probin
--------------------+-----------------------------------------------
 gpcc_filenode_scan | /opt/greenplum-cc-web-4.9.0/lib/gp_browser.so

we can run the below query on each DB: 

# \c <DB_NAME>
# SELECT gp_segment_id, (select nspname from pg_namespace where oid = pronamespace) as namespace ,proname as name,probin from gp_dist_random('pg_proc') where probin::text ~ 'gp_browser.so';

To avoid such an issue happening during gp_upgrade, the customer should be advised to run gpcheckcat before upgrading the DB, it should be run against all Database to make sure there are no orphan functions. as the document requested:
https://docs.vmware.com/en/VMware-Tanzu-Greenplum-Upgrade/1.3/greenplum-upgrade/GUID-gpupgrade_preupgrade.html