RHEL8.9 'libzstd' package compatibility issue with Greenplum 6.26.0 and below
Article ID: 296937
Updated On:
Products
Issue/Introduction
Summary
After the change in upstream image for el8 wherein the version got changed to 8.9 , some of the gpdb utilities are throwing error. These errors are observed in greenplum clusters with version 6.26.0 and below. The customers will encounter the issue if they either upgrade from el7 to el8 or if they do yum update on el8 machine(which gets updated to latest version i.e 8.9)
Symptom
When we upgrade the operating system to a certain release of RHEL or RHEL-like OS (for example 8.9+), If we load(# source) the greenplum_path.sh file, we might notice that some commands like rpm , gppkg, gpinitsystem etc will no longer work
For example:
# source /usr/local/greenplum-db-6.22.0/greenplum_path.sh # rpm rpm: symbol lookup error: /lib64/libelf.so.1: undefined symbol: ZSTD_compressStream2 # ip ip: symbol lookup error: /lib64/libelf.so.1: undefined symbol: ZSTD_compressStream2
20231205:14:34:40:023228 gpsegstart.py_utlxa454:gpadmin:utlxa454:gpadmin-[CRITICAL]:-Error occurred: non-zero rc: -11 Command was: '$GPHOME/bin/pg_controldata /data1/primary/gpseg17' rc=-11, stdout='', stderr=''
Environment
Resolution
Root Cause Analysis:
The reason for failure is because greenplum_path.sh file will set the OS environment parameter $LD_LIBRARY_PATH, with the path under $GPHOME
echo $LD_LIBRARY_PATH /usr/local/greenplum-db-6.22.0/lib:/usr/local/greenplum-db-6.22.0/ext/python/lib:/opt/greenplum_6.22.0/lib:/opt/greenplum_6.22.0/ext/python/lib:/opt/greenplum_6.22.0/ext/python3.9/lib:/usr/local/greenplum-db-6.22.0/ext/python3.9/lib
The libzstd.so under $GPHOME conflicts with the OS lib file: the other OS lib file needs to use function ZSTD_compressStream2 but the libzstd in $GPHOME does not have such a function.
See below for example:
### The OS libzstd lib file:
# ll /lib64/libzstd.so.1
lrwxrwxrwx. 1 root root 16 Apr 12 2021 /lib64/libzstd.so.1 -> libzstd.so.1.4.4
# readelf -s /lib64/libzstd.so.1 | grep ZSTD_compressStream2
61: 0000000000011c70 131 FUNC GLOBAL DEFAULT 12 ZSTD_compressStream2_simp
181: 0000000000011390 2210 FUNC GLOBAL DEFAULT 12 ZSTD_compressStream2
### the Greenplum provided libzstd lib file:
# ll $GPHOME/lib/libzstd.so.1
lrwxrwxrwx 1 gpadmin gpadmin 16 Sep 10 2022 /opt/greenplum_6.22.0/lib/libzstd.so.1 -> libzstd.so.1.3.7
# readelf -s $GPHOME/lib/libzstd.so.1 | grep ZSTD_compressStream2
<no output>
Resolution：
- On all hosts, update the greenplum_path.sh file , add “export LD_LIBRARY_PATH=“ at the end of the file and source the file. this will unset the LD_LIBRARY_PATH and take system libzstd priority over greenplum/lib
echo “export LD_LIBRARY_PATH=“ >> greenplum_path.sh source greenplum_path.sh
- Run gppkg / gpinitsystem / any utility which is giving the failure for libzstd.
- Once the utility execution is successful revert/remove the "export LD_LIBRARY_PATH= " from greenplum_path.sh to run GPDB.
Fix：
The issue has been fixed under 6.26.1 and afterward, see the release note for more details https://docs.vmware.com/en/VMware-Greenplum/6/greenplum-database/relnotes-release-notes.html Zstandard (zstd) Supported Version This release updates the supported zstd version on RHEL 8 to 1.4.4
Feedback
