Backup Failure Due to Directory Creation Issues in Greenplum Backup Utility (gpbackup)
Article ID: 406336
Updated On:
Products
Issue/Introduction
A backup attempt using the gpbackup utility might fail when specifying a custom backup directory using the --backup-dir option. The backup process would be successful when the --backup-dir option is omitted.
[CRITICAL]:-Unable to create backup directories on <segID> segment. See /home/gpadmin/gpAdminLogs/<gpbackup.log> for a complete list of errors.
[WARNING]:-Encountered error during cleanup: runtime error: invalid memory address or nil pointer dereference
This indicates that the specified backup directory could not be created on one of the segments, leading to the failure during the cleanup phase.
Environment
All GPDB versions
Cause
The issue occurrs because the specified backup directory could not be created on the Greenplum segments. The failure during the cleanup phase was likely triggered by one of the following causes:
-
Disk Mount Issue: The backup path may not be mounted correctly or may not be accessible by the gpadmin user on the segment nodes. If the backup directory is on a separate disk or an NFS mount, it is important to ensure that it is properly mounted on all segment nodes and is accessible for writing.
-
Permissions Issue: The gpadmin user may not have the necessary read and write permissions to create directories or write data in the specified backup directory.
-
Disk Space/Quota Issues: If the disk is full or there is an inode exhaustion on the backup directory, it can prevent the backup process from creating the required directories and files, resulting in errors during the backup process.
-
Incorrect Directory Path: There could be a typo or incorrect path used for the backup directory, which could lead to an inability to create the backup directory.
Resolution
a) Ensure that the backup directory is mounted correctly and is accessible from all segment hosts. Run the following command on all segment hosts to check the mount status:
df -h <directory_path>
b) Verify the permissions for the backup directory. The gpadmin user should have both read and write access to the directory
ls -ld <directory_path>
c) Ensure that there is enough disk space and available inodes on the backup directory.
df -h <directory_path> # Check disk space
df -i <directory_path> # Check inodes usage
d) As the gpadmin user, manually test creating a directory in the backup path on each segment host to ensure it can write to the location.
mkdir <directory_path>/testdirFeedback
