Installation of pg_crypto fails with "ERROR: could not load library "xxxx/pgcrypto.so": xxxx/lib/postgresql/pgcrypto.so: undefined symbol: FIPS_mode_set"

Products

• VMware Tanzu Greenplum

Issue/Introduction

Symptoms:

Installation of pgcrypto fails with the following error message:

[gpadmin@gpdbsne lib64]$ psql -d <database-name> -f $GPHOME/share/postgresql/contrib/pgcrypto.sql
SET
psql:/usr/local/greenplum-db/./share/postgresql/contrib/pgcrypto.sql:9: ERROR: could not load library "/usr/local/greenplum-db-4.3.5.3/lib/postgresql/pgcrypto.so": /usr/local/greenplum-db-4.3.5.3/lib/postgresql/pgcrypto.so: undefined symbol: FIPS_mode_set
[......]
[......] 

Environment

Cause

The issue is caused by the wrong crypto libraries being referenced.
 

For example, the crypto library "libcrypto.so.0.9.8" references the crypto library in the "greenplum-perfmon-web-4.1.1.5-build-1" installation directory. However, it should be referencing the Greenplum database libraries.

[gpadmin@gpdbsne lib64]$ ldd /usr/local/greenplum-db-4.3.5.3/lib/postgresql/pgcrypto.so
	linux-vdso.so.1 =>  (0x00007fffa01ff000)
	libcrypto.so.0.9.8 => /usr/local/greenplum-perfmon-web-4.1.1.5-build-1/lib/libcrypto.so.0.9.8 (0x00007f3a9cd21000)
	libz.so.1 => /usr/local/greenplum-cc-web/./lib/libz.so.1 (0x00007f3a9cb0a000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f3a9c75e000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f3a9c55a000)
	/lib64/ld-linux-x86-64.so.2 (0x0000003601e00000) 

This is caused by the method used to set the environment variables when the user connects to the Unix server. For example, in this case, the LD_LIBRARY_PATH environment variable is set.

[gpadmin@gpdbsne lib64]$ env | grep LD
LD_LIBRARY_PATH=/usr/local/greenplum-perfmon-web-4.1.1.5-build-1/lib:/usr/local/greenplum-cc-web-1.3.0.3-build-18/lib:/usr/local/greenplum-db/./lib:/usr/local/greenplum-db/./ext/python/lib:   

As shown in the setting above, the following locations "/usr/local/greenplum-perfmon-web-4.1.1.5-build-1/lib" & "/usr/local/greenplum-cc-web-1.3.0.3-build-18/lib" are used to reference the crypto libraries first, This is caused by the method used to set up the environment path in the .bashrc file (user updated).

Note: The last line of the source always uses the first batch to check the path of the libraries.

source /usr/local/greenplum-db/greenplum_path.sh
source /usr/local/greenplum-db/gpcc_path.sh
source /usr/local/greenplum-db/gpperfmon_path.sh 

The issue is that the postmaster of the database points to the crypto file of two different installations and not the Greenplum database.

[gpadmin@gpdbsne lib64]$ lsof -p 31949  | grep crypto
postgres 31949 gpadmin  mem    REG              253,1   309353 1191079 /usr/local/greenplum-cc-web-1.3.0.3-build-18/lib/libk5crypto.so.3.1
postgres 31949 gpadmin  mem    REG              253,1  1788415 1325183 /usr/local/greenplum-perfmon-web-4.1.1.5-build-1/lib/libcrypto.so.0.9.8
[gpadmin@gpdbsne lib64]$ 

Resolution

Follow the steps to resolve this issue:

1. Correct the environment to have the Greenplum library sourced at the end:

   source /usr/local/greenplum-db/gpcc_path.sh
   source /usr/local/greenplum-db/gpperfmon_path.sh
   source /usr/local/greenplum-db/greenplum_path.sh 
   

2. After refreshing, observe that the path now points to the correct location.

   [gpadmin@gpdbsne lib64]$ env | grep LD
   LD_LIBRARY_PATH=/usr/local/greenplum-db/./lib:/usr/local/greenplum-db/./ext/python/lib:/usr/local/greenplum-perfmon-web-4.1.1.5-build-1/lib:/usr/local/greenplum-cc-web-1.3.0.3-build-18/lib: 
   

3. Restart the database and the new postmaster process will now point to the correct libraries.

   [gpadmin@gpdbsne lib64]$ lsof -p 19896 | grep crypto
   postgres 19896 gpadmin  mem    REG              253,1   583652 1071088 /usr/local/greenplum-db-4.3.5.3/lib/libk5crypto.so.3.1
   postgres 19896 gpadmin  mem    REG              253,1  1855858 1071063 /usr/local/greenplum-db-4.3.5.3/lib/libcrypto.so.0.9.8
   [gpadmin@gpdbsne lib64]$ 
   

4. Re-execute the pgcrypto file again using the command below:

   psql -d <database-name> -f $GPHOME/share/postgresql/contrib/pgcrypto.sql