gpinitstandby Fails with Error "pg_basebackup: could not read COPY data: server closed the connection unexpectedly" due to Lack of Disk Space

Products

• VMware Tanzu Greenplum

Issue/Introduction

Symptoms:

During the standby initialization, you encounter the following error message:

gpinitstandby:mdw:gpadmin-[ERROR]:-Error initializing standby master: ExecutionError: 'non-zero rc: 1' occured. Details: 'ssh -o 'StrictHostKeyChecking no' smdw ". /usr/local/greenplum-db/./greenplum_path.sh; pg_basebackup -x -R -E ./pg_log -E ./gp_dumps -E ./gpperfmon/data -D /data/master/gpseg-1 -h mdw -p 5432"' cmd had rc=1 completed=True halted=False
stdout=''
stderr='pg_basebackup -x -R -E ./pg_log -E ./gp_dumps -E ./gpperfmon/data -D /data/master/gpseg-1 -h mdw -p 5432
pg_basebackup: could not read COPY data: server closed the connection unexpectedly

This probably means that the server terminated abnormally before or while processing the request.

 

Environment

Cause

It was observed that the master data directory ($MASTER_DATA_DIRECTORY) on the master host was of the size greater than the size of the disk on the standby master (the directory on which the copy attempt is made), thus causing the copy to fail.

 

Resolution

• In this instance, there was a big script file that was occupying the space. Moving it out of the master data directory and attempting to re-initialise the standby, helped to overcome the problem. So, move any unrelated database files out of the master data directory and into the master host.
• Ensure there is enough disk space (i.e., greater than the size of the master data directory on the master host).