Backup from gpcrondump to Netbackup Fails with "error while loading shared libraries"

Products

• VMware Tanzu Greenplum

Issue/Introduction

Symptoms:

When backing up with gpcrondump to Netbackup, it fails with errors in the master.

Error Message

In the .rpt files, we can see the following error:

Master (dbid 1) Host mdw Port 5432 Database dbtest BackupFile /data/master/gpseg-1/db_dumps/20170628104103/dbtest_test_gp_dump_-1_1_20170628104103_post_data: Failed with error:
{ 20170628:11:05:06|gp_dump_agent-[ERROR]:-could not write to output file: Broken pipe

The following can be seen in pg_log on the master:

2017-06-28 11:04:07.084905 CEST,,,p7996,th1489581856,,,,0,,,seg-1,,,,,"LOG","00000","3rd party error log:
/usr/local/greenplum-db/./bin/gp_bsa_dump_agent: error while loading shared libraries: libgpbsa.so: cannot open shared object file: No such file or directory",,,,,,,,"SysLoggerMain","syslogger.c",550,

 

Environment

Cause

The variable LD_LIBRARY_PATH might contain out of date lib paths.

This documentation indicates we should add the following path to the variable LD_LIBRARY_PATH

$GPHOME/lib/nbuNN/lib:$LD_LIBRARY_PATH

However, if there is any other path listed before this, it might be causing conflicts if it is pointing to an outdated lib version, for example, /usr/local/greenplum-cc-web/./lib.

 

Resolution

A permanent solution would be to upgrade any out of date components that might be causing the issue, in this case, GPCC from 3.0.1 to 3.2.2.

A workaround or a way to test the issue would be to list the conflicting paths at the end of the variable LD_LIBRARY_PATH and restart the database. The backup should succeed.