gpexpend fails with "gpexpand_schema.tar: No space left on device"
search cancel

gpexpend fails with "gpexpand_schema.tar: No space left on device"

book

Article ID: 295829

calendar_today

Updated On:

Products

VMware Tanzu Greenplum

Issue/Introduction

Symptoms:

While running gpexpand in Greenplum Database (GPDB), it fails with the error message: stderr='scp: ./gpexpand_schema.tar: No space left on device'.

Error Message

20151103:10:45:49:182997 gpexpand:mdw:gpadmin-[INFO]:-Adding new segments into template pg_hba.conf
20151103:10:45:49:182997 gpexpand:mdw:gpadmin-[INFO]:-Creating schema tar file
20151103:10:51:00:182997 gpexpand:mdw:gpadmin-[INFO]:-Distributing template tar file to new hosts
20151103:10:56:08:182997 gpexpand:mdw:gpadmin-[ERROR]:-gpexpand failed: ExecutionError: 'Error Executing Command: ' occured. Details: 'GPSTART_INTERNAL_MASTER_ONLY=1 /usr/bin/scp -o 'StrictHostKeyChecking no' -r gpexpand_schema.tar sdw6.gphd.local:.' cmd had rc=1 completed=True halted=False
stdout=''
stderr='scp: ./gpexpand_schema.tar: No space left on device
' 
Exiting...
20151103:10:56:08:182997 gpexpand:mdw:gpadmin-[ERROR]:-Please run 'gpexpand -r -D dca_expansion_db' to rollback to the original state.
20151103:10:56:08:182997 gpexpand:mdw:gpadmin-[INFO]:-Shutting down gpexpand...

Attempting to start the database results in the following error:

    gpstart:mdw:gpadmin-[ERROR]:-gpstart error: Found a System Expansion Setup in progress. Please run 'gpexpand --rollback'

    Note: The database needs to be running to roll back.

    gpexpand:mdw:gpadmin-[ERROR]:-gpexpand failed: could not connect to server: Connection refused
        Is the server running on host "localhost" (127.0.0.1) and accepting
        TCP/IP connections on port 5432?
Exiting...

    Environment


    Cause

    gpexpand tars the entire MASTER_DATA_DIRECTORY. This includes the backup directory db_dumps and log directory pg_log. This can create a massive uncompressed file. 

    gpexpand generates gp_expandschema.tar in the directory where gpexpand is executed.

    It will secure copy (SCP) the tar to the same directory on the segments, unless the --tardir flag is used to specify the directory.

    If gpexpand is executed in a directory that does not have enough free space on the master or any segment, gpexpand will fail.

    Resolution

    Follow the below instructions to resolve this issue: 

    1. Start the database in restricted mode and roll back the gpexpand. 
    $ gpstart -aR
$ gpexpand -r -D testdb
    2. Move the backup directory and log directory from MASTER_DATA_DIRECTORY. Usually /data has enough space and you can move it one level up to remove it from the tar file.

    3. Check the size of the MASTER_DATA_DIRECTORY and the available space of the filesystem. 
    $ du -hs $MASTER_DATA_DIRECTORY
1.1G    /data/master/gpseg-1/

$ df -h
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/rhel-root   35G   28G  7.2G  80% /
devtmpfs               1.9G     0  1.9G   0% /dev
tmpfs                  1.9G     0  1.9G   0% /dev/shm
tmpfs                  1.9G   60M  1.8G   4% /run
tmpfs                  1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/sda1             1014M  197M  818M  20% /boot
tmpfs                  378M     0  378M   0% /run/user/0
tmpfs                  378M     0  378M   0% /run/user/1002
    4. Confirm there is enough free space on the segments using the following command:  
    $ gpssh -f hostfile -e "df -h [destination directory]"
    5. Execute gpexpand from a location that has sufficient free space to build gpexpand_schema.tar. There should be more free space than the size of MASTER_DATA_DIRECTORY.

    Powered by Wolken Software