GPDB 6.28.x: System-wide Query Failure and Core Dumps due to R_HOME and LD_LIBRARY_PATH Mismatch

Products

• VMware Tanzu Greenplum
• VMware Tanzu Greenplum / Gemfire

Issue/Introduction

Queries calling PL/R User Defined Functions (UDFs) crash repeatedly with segment PANIC / Signal 11 (Segmentation fault).

The database log displays messages similar to:

LOG: server process (PID xxx) was terminated by signal 11: Segmentation fault
Failed process was running: SELECT ... unnest(plr_function(...))

The database log displays messages similar to:

LOG: server process (PID xxx) was terminated by signal 11: Segmentation fault
Failed process was running: SELECT ... unnest(plr_function(...))

GDB stack traces from core dumps show crashes inside R runtime functions, such as:

#0  0x00007fe596ca349b in raise ()
#1  0x0000000000c0b19c in StandardHandlerForSigillSigsegvSigbus_OnMainThread ()
#2  <signal handler called>
#3  0x00007fe566088f26 in Rf_installTrChar () from /usr/lib64/R/lib/libR.so

Cause

An ABI (Application Binary Interface) mismatch occurs when LD_LIBRARY_PATH and R_HOME point to two different R versions across segment nodes.

For example, if a custom or OS-level R version (such as R 3.6) is prepended to LD_LIBRARY_PATH, while R_HOME points to Greenplum's embedded R version (R 3.3):

1. The dynamic linker loads the R 3.6 runtime library (libR.so) via LD_LIBRARY_PATH.

2. libR.so then attempts to load R package libraries (such as methods.so) from $R_HOME (R 3.3).

3. Starting in R 3.5, internal memory layout for array objects changed. When R 3.6 engine functions process memory mapped by R 3.3 compiled libraries, incorrect memory offset calculations trigger an invalid memory access (SIGSEGV).

Resolution

 

• Step 1: Check Environment Variables Across All Nodes Inspect .bash_profile, .bashrc, and system environment files on all host nodes (Coordinator and Segments) to see if external R paths are set in LD_LIBRARY_PATH or R_HOME.

• Step 2: Remove External R Paths Remove any hardcoded paths pointing to non-Greenplum R installations (e.g., /usr/lib64/R or /usr/local/lib64/R) from LD_LIBRARY_PATH and PATH.

• Step 3: Standardize R Environment Variables Ensure all nodes use Greenplum's embedded R directory structure. Verify that $GPHOME/greenplum_path.sh includes the standard PL/R environment definitions:

  export R_HOME=$GPHOME/ext/R-3.3.3
  export LD_LIBRARY_PATH=$R_HOME/lib:$R_HOME/extlib:$LD_LIBRARY_PATH
  export PATH=$R_HOME/bin:$PATH

• Step 4: Restart the Cluster Restart the Greenplum Database cluster to apply clean environment variables across all segment processes:

  gpstop -M fast
  gpstart