GPCC installation failed with error "Error when copying binary to xxx"

Products

• VMware Tanzu Greenplum

Issue/Introduction

When installing the Greenplum Command Center(GPCC), we execute the installation package on the master. then the master will pack the binary files to /tmp/gpcc.tar.gz, and distribute them to all hosts. 
The distribution is done by the linux command "scp", if the scp command failed, the installation script will report error like: "Error when copying binary to xxx"

The failure of the scp could be due to various reasons, with either a network issue or an OS issue. 
This article will show you how to troubleshoot this. 

Environment

Product Version: 6.23

Resolution

Usually, the error message from the installation is like below:

CREATING SUPERUSER 'gpmon'...
this is some text from ~/.bashrc
CREATING COMMAND CENTER DATABASE 'gpperfmon'...
RELOADING pg_hba.conf. PLEASE WAIT ...
...
RunCommandOnEachHost failed on host: mdw
Error when copying binary to mdw this is some text from ~/.bashrc
Error when copying binary to support-gpddb6-sdw02
Error when copying binary to support-gpddb6-sdw01
CopyBinaryToHost error: RunCommandOnEachHost failure happened

As a first step, we need to make sure the temp file on the master was generated properly, run the below command and we expect the return code of tar to be 0 and tar command should list all files without error 

# tar -ztvf /tmp/gpcc.tar.gz
# echo $?

Once confirmed the local tar file is good, test the scp command on each host, we can use the below script: (change the name of the log file accordingly)

log_file="gpccinstall.2023-05-30_124529.log"

for host in `cat $log_file | grep "Error when copying binary to" | awk -F'Error when copying binary to' '{print $2}' | awk '{print $1}'`; do echo "checking [$host]..."; /bin/bash -c "scp -o StrictHostKeyChecking=no /tmp/gpcc.tar.gz ${host}:/tmp"; echo return code is [$?] ; done

Review the output:

    checking [mdw]...
    this is some text from ~/.bashrc
    return code is [1]
    checking [support-gpddb6-sdw02]...
    gpcc.tar.gz                       
    return code is [0]
    checking [support-gpddb6-sdw01]...   
    gpcc.tar.gz                       
    return code is [0]

in the above example, we see the scp to mdw (master itself) failed. it did not start the transmission, instead, it just shows some strange message.  
if that is the case, please check if we have any "echo" command in ~/.bashrc  

$ cat ~/.bashrc | grep echo
echo "this is some text from ~/.bashrc"

According to Redhat, the echo in the .bashrc file will break the scp command. so the workaround for this issue is to remove all echo commands in the ~/.bashrc file  
refer to: https://bugzilla.redhat.com/show_bug.cgi?id=20527