gpupgrade Finalize Fails with psql Connection Error to "/var/run/postgresql/.s.PGSQL.5432"

Products

VMware Tanzu Data Suite

Issue/Introduction

During a Greenplum Database upgrade using gpupgrade, the process fails specifically during the finalize stage. The upgrade logs or terminal output will display the following psql connection error:

psql: error: could not connect to server: No such file or directory 
Is the server running locally and accepting connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"

Environment

Environment & Prerequisites

This issue occurs when all three of the following conditions are met in the environment:

PL/Java is installed: The PL/Java extension has been installed on the target cluster, which generates the configuration file ${GPHOME}/etc/environment.d/30-pljava.conf.
OS PostgreSQL is installed: An open-source, OS-level version of PostgreSQL exists on the system (e.g., installed via yum or apt), meaning OS-level utilities like /bin/psql are present.
Missing Global Java Path: The JAVA_HOME environment variable is not defined globally for the operating system user.

Cause

Root Cause Analysis

This failure is caused by PATH variable pollution triggered by the PL/Java extension configuration interacting with the gpupgrade process.

The Mechanism of Failure:

As per standard PL/Java installation guidelines, administrators manually set JAVA_HOME directly inside greenplum_path.sh on the source cluster.
During the upgrade, gpupgrade generates a new greenplum_path.sh file for the target 7.x cluster. However, this new script does not inherit or copy the JAVA_HOME variable from the source script.
If PL/Java is installed on the target cluster during the initialize stage, it creates the 30-pljava.conf file, which contains the following logic: export PATH=$JAVA_HOME/bin:$PATH
Because JAVA_HOME is empty in the newly generated target script, the shell evaluates the command as: export PATH=/bin:$PATH
This places the system /bin directory at the absolute front of the user's PATH.
Consequently, when the script attempts to run database commands, the system executes the OS-level /bin/psql instead of the Greenplum binary. The OS-level psql looks for the socket in the default open-source location (/var/run/postgresql) rather than Greenplum's configured location (/tmp), resulting in the connection failure.

Resolution

Workaround

When using gpupgrade, if you need to install the PL/Java package on the target cluster, you must explicitly configure the Java environment for the new cluster.

To bypass this issue, strictly follow Steps 1 through 6 from the official PL/Java extension guide on the target cluster before proceeding:

Reference: Installing the Greenplum PL/Java Extension

Key Requirement: You must ensure that the environment variables JAVA_HOME and LD_LIBRARY_PATH are properly set directly within the $GPHOME/greenplum_path.sh file across all Greenplum Database hosts. (On target cluster)

Permanent Resolution

PLJAVA Package Update (Target Release: 2.0.10): The PL/Java package will update the 30-pljava.conf file to include a safety check. It will verify if JAVA_HOME exists; if it is missing, the script will output a warning and safely bypass modifying the PATH and LD_LIBRARY_PATH variables.

if [ -z "$JAVA_HOME" ]; then
    echo "Warning: 'JAVA_HOME' is not set. pljava won't work without it." 1>&2
else
    # for jdk 8
    LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/amd64/server:$LD_LIBRARY_PATH
    # for jdk 11
    LD_LIBRARY_PATH=$JAVA_HOME/lib/server:$LD_LIBRARY_PATH
    export LD_LIBRARY_PATH
    export PATH=$JAVA_HOME/bin:$PATH
fi