How to delete data and re-claim space from Data Domain System using gpddboost and NFS exports
search cancel

How to delete data and re-claim space from Data Domain System using gpddboost and NFS exports

book

Article ID: 295847

calendar_today

Updated On:

Products

VMware Tanzu Greenplum

Issue/Introduction

Symptoms:

When a user tries to delete data from the Data Domain system and re-claim cleanable space using the command "gpcrondump -o --ddboost", the following error is produced:

20130827:15:26:50|ddboost-[ERROR]:-ddboost create storage unit failed. Err = 5040
20130827:15:26:50|gpddboost-[ERROR]:-Could not connect to DD_host with DD_user and the DD_password.
20130827:15:26:53:008209 gpcrondump:dca1_mastr1:gpadmin-[ERROR]:-gpcrondump error: Failed to connect to DD_host with DD_user and the DD_password

Environment


Cause

The message indicating "failed to connect to Data Domain with the username/password" is misleading.

Obserbe the error message "[ERROR]:-ddboost create storage unit failed". This provides the hint that it is not able to create the needed storage unit. A quick check on the Data Domain server shows that the filesystem useage is 100%. See the explanation below:

gpddb_user@DD640XIAE> filesys show space
Active Tier:
Resource           Size GiB   Used GiB   Avail GiB   Use%   Cleanable GiB
----------------   --------   --------   ---------   ----   -------------
/data: pre-comp           -   290616.1           -      -               -
/data: post-comp     7580.7     7580.7         0.0   100%             0.0
/ddvar                 78.7        2.2        72.6     3%               -
----------------   --------   --------   ---------   ----   -------------

Resolution

There are two ways to resolve this issue and reclaim space on Data Domain:
 

  • Free space on the Data Domain (unless more disk can be added)
  • Schedule cleanup activities on the Data Domain


From DCA or GPDB software, using "gpddboost" clear up some space

1. See what the default backup directory is using: gpddboost --show-config 
[gpadmin@dca1_mastr1 ~]$ gpddboost --show-config
20130827:18:47:52|ddboost-[DEBUG]:-Libraries were loaded successfully
20130827:18:47:52|ddboost-[INFO]:-opening LB on /home/gpadmin/DDBOOST_CONFIG
Data Domain Hostname:dd640_1_1
Data Domain Boost Username:gpddb_user
Default Backup Directory:db_dumps
Data Domain default log level:WARNING
2. List the directory under db_dumps using the command below:  
[gpadmin@dca1_mastr1 ~]$ gpddboost --listDirectory  --dir=db_dumps
20130827:18:48:07|ddboost-[DEBUG]:-Libraries were loaded successfully
20130827:18:48:07|ddboost-[INFO]:-opening LB on /home/gpadmin/DDBOOST_CONFIG
20130827:18:48:07|gpddboost-[DEBUG]:-ddp_open_dir(db_dumps)
20130227
20130301
20130311
20130312
[.....]
20130825
3. Use the following command to delete the directory: 
gpddboost --del-dir=db_dumps/20130618

Note: Only delete the directories that are safe to delete.
 

4. Verify Cleanable Gib space. 
sysadmin@DD640XIAE# filesys show space
Active Tier:
Resource           Size GiB   Used GiB   Avail GiB   Use%   Cleanable GiB
----------------   --------   --------   ---------   ----   -------------
/data: pre-comp           -   154789.5           -      -               -
/data: post-comp     7580.7     7511.2        69.5    99%          2038.5
/ddvar                 78.7        2.2        72.6     3%               -
----------------   --------   --------   ---------   ----   -------------

Note: Cleanable Gib indicates the space that can be reclaimed with the garbage collection process.
 

5. After deleting the directories, run the command after logging on DD as a sysadmin to make the free space available. 
'filesys clean start nowait'

Note: This initiates the garbage collection process.
 

6. Watch the status using filesys clean watch or filesys clean status. It will take around 4-5 hours, but it depends on the size. 
sysadmin@DD640XIAE# filesys clean status
Cleaning started at 2013/08/27 06:00:02: phase 9 of 10 (copy)
 67.1% complete,     0 GiB free; time: phase  0:09:11, total  3:32:50
7. View the current space usage using after the file sys clean is completed. 
sysadmin@DD640XIAE# filesys show space

Login to Data Domain, export the NFS directory and mount the backup directory of DD on DCA and then delete the directories:

8. Verify the "mtree" list after logging on to DCA. 
sysadmin@DD640XIAE# mtree list
Name                Pre-Comp (GiB)   Status
-----------------   --------------   ------
/data/col1/GPDB           133508.8   RW
/data/col1/backup              0.0   RW
-----------------   --------------   ------
9. Setup NFS export on Data Domain for mtree /data/col1/GPDB. 
sysadmin@DD640XIAE# nfs add /data/col1/GPDB   *
Hint: * is to allow all client, but you could mention IP like 172.10.10.0/24 as well.
sysadmin@DD640XIAE# nfs show clients
path              client          options
---------------   -------------   ----------------------------------------
/data/col1/GPDB   *               (rw,no_root_squash,no_all_squash,secure)

[root@dca1_mastr1 ~]#  
mkdir /bkp_pivGPDB; 
mount -t nfs -o hard,intr,nfsvers=3,tcp,rsize=32768,wsize=32768 dd640_1_1:/data/col1/GPDB /bkp_pivGPDB
10. Use the following Linux commands on the shared filesystem to delete the data on the Data Domain. 
rm -rf /bkp_pivGPDB/db_dumps/20130620  
[root@dca1_mastr1 ~]# ls /bkp_pivGPDB/
db_dumps
[root@dca1_mastr1 ~]# ls /bkp_pivGPDB/db_dumps/
20130620  20130624  20130628  20130713  20130717  20130721  20130725  20130729  20130802
11. After deleting the directories, log on to the Data Domain as sysadmirun and the following command to make the free space available: 
filesys clean start nowait
12. Watch the status using the following commands on the Data Domain: 
filesys clean status or filesys clean status

Note: The garbage collection process could take around 4-5 hours for around 10 TB of total disk space depending on the size.


Powered by Wolken Software