gpdbrestore using "--truncate option" Combined with --table-file option fails with error message "CRITICAL:-gpdbrestore failed."
Article ID: 295703
Updated On:
Products
VMware Tanzu Greenplum
Issue/Introduction
Symptoms:
While restoring from the backup using --truncate option combined with --table-file option, if the dump file contains a new table that is not present in the target database, and the new table's name is present in the file, the process aborts with an error:
" CRITICAL:-gpdbrestore failed. (Reason=''str' object is not callable') exiting..."
Environment
Cause
Important parameters involved in the restore command-
--truncate - Truncate table data before restoring data to the table from the backup. If this option is not specified, existing table data is not removed before data is restored to the table.This option is supported only when restoring a set of tables with the option -T or --table-file.This option is not supported with the -e option.
--table-file - Specify a file that contains a list of table names to restore. The file contains any number of table names, listed one per line.
When --truncate option is specified, a truncate statement is generated for each table listed in the --table-file. When these statements are executed, if a table specified in the --table-file does not exist currently in the destination database, an error is generated.
Example:
File "tables.txt" contains:
public.t1
public.t2
public.t2
Table public.t2 does not exist in destination database (the database where data is restored).
Command "gpdbrestore -t <key> --truncate --table-file tables.txt" generates truncate commands and fails when the statement "TRUNCATE TABLE public.t2;" is attempted to be executed.
Resolution
1. Create Database
template1=# create database fpl_test; CREATE DATABASE
2. Connect to it and check the version
template1=# \c fpl_test You are now connected to database "fpl_test" as user "gpadmin". fpl_test=# select version(); version ------------------------------------------------------------------------------------------------------------------------------------------------------ PostgreSQL 8.2.15 (Greenplum Database 4.3.5.1 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on May 14 2015 14:07:14 (1 row)
3. Create Schema
fpl_test=# create schema fpl; CREATE SCHEMA
4. Create test tables to be backed up
fpl_test=# create table fpl.x1(a int); NOTICE: Table doesn't have 'DISTRIBUTED BY' clause -- Using column named 'a' as the Greenplum Database data distribution key for this table. HINT: The 'DISTRIBUTED BY' clause determines the distribution of data. Make sure column(s) chosen are the optimal data distribution key to minimize skew. CREATE TABLE fpl_test=# create table fpl.x2(a int); NOTICE: Table doesn't have 'DISTRIBUTED BY' clause -- Using column named 'a' as the Greenplum Database data distribution key for this table. HINT: The 'DISTRIBUTED BY' clause determines the distribution of data. Make sure column(s) chosen are the optimal data distribution key to minimize skew. CREATE TABLE fpl_test=# create table fpl.x3(a int); NOTICE: Table doesn't have 'DISTRIBUTED BY' clause -- Using column named 'a' as the Greenplum Database data distribution key for this table. HINT: The 'DISTRIBUTED BY' clause determines the distribution of data. Make sure column(s) chosen are the optimal data distribution key to minimize skew. CREATE TABLE fpl_test=# create table fpl.x4(a int); NOTICE: Table doesn't have 'DISTRIBUTED BY' clause -- Using column named 'a' as the Greenplum Database data distribution key for this table. HINT: The 'DISTRIBUTED BY' clause determines the distribution of data. Make sure column(s) chosen are the optimal data distribution key to minimize skew. CREATE TABLE fpl_test=# \dt List of relations Schema | Name | Type | Owner | Storage --------+------+-------+---------+--------- public | x1 | table | gpadmin | heap public | x2 | table | gpadmin | heap public | x3 | table | gpadmin | heap public | x4 | table | gpadmin | heap (4 rows) fpl_test=# \q
5. Take full database backup
[gpadmin@mdw ~]$ gpcrondump -x fpl_test 20151028:11:33:51:006542 gpcrondump:mdw:gpadmin-[INFO]:-Starting gpcrondump with args: -x fpl_test 20151028:11:33:51:006542 gpcrondump:mdw:gpadmin-[INFO]:---------------------------------------------------- 20151028:11:33:51:006542 gpcrondump:mdw:gpadmin-[INFO]:-Master Greenplum Instance dump parameters 20151028:11:33:51:006542 gpcrondump:mdw:gpadmin-[INFO]:---------------------------------------------------- 20151028:11:33:51:006542 gpcrondump:mdw:gpadmin-[INFO]:-Dump type = Full database 20151028:11:33:51:006542 gpcrondump:mdw:gpadmin-[INFO]:-Database to be dumped = fpl_test 20151028:11:33:51:006542 gpcrondump:mdw:gpadmin-[INFO]:-Master port = 8004 20151028:11:33:51:006542 gpcrondump:mdw:gpadmin-[INFO]:-Master data directory = /data/master/eddie_4351-1/ 20151028:11:33:51:006542 gpcrondump:mdw:gpadmin-[INFO]:-Run post dump program = Off 20151028:11:33:51:006542 gpcrondump:mdw:gpadmin-[INFO]:-Rollback dumps = Off 20151028:11:33:51:006542 gpcrondump:mdw:gpadmin-[INFO]:-Dump file compression = On 20151028:11:33:51:006542 gpcrondump:mdw:gpadmin-[INFO]:-Clear old dump files = Off 20151028:11:33:51:006542 gpcrondump:mdw:gpadmin-[INFO]:-Update history table = Off 20151028:11:33:51:006542 gpcrondump:mdw:gpadmin-[INFO]:-Secure config files = Off 20151028:11:33:51:006542 gpcrondump:mdw:gpadmin-[INFO]:-Dump global objects = Off 20151028:11:33:51:006542 gpcrondump:mdw:gpadmin-[INFO]:-Vacuum mode type = Off 20151028:11:33:51:006542 gpcrondump:mdw:gpadmin-[INFO]:-Ensuring remaining free disk > 10 Continue with Greenplum dump Yy|Nn (default=N): > y 20151028:11:33:56:006542 gpcrondump:mdw:gpadmin-[INFO]:-Directory /data/master/eddie_4351-1/db_dumps/20151028 not found, will try to create 20151028:11:33:56:006542 gpcrondump:mdw:gpadmin-[INFO]:-Created /data/master/eddie_4351-1/db_dumps/20151028 20151028:11:33:56:006542 gpcrondump:mdw:gpadmin-[INFO]:-Checked /data/master/eddie_4351-1/ on master 20151028:11:33:57:006542 gpcrondump:mdw:gpadmin-[INFO]:-Configuring for single database dump 20151028:11:33:57:006542 gpcrondump:mdw:gpadmin-[INFO]:-Validating disk space 20151028:11:33:58:006542 gpcrondump:mdw:gpadmin-[INFO]:-Adding compression parameter 20151028:11:33:58:006542 gpcrondump:mdw:gpadmin-[INFO]:-Adding --no-expand-children 20151028:11:33:58:006542 gpcrondump:mdw:gpadmin-[INFO]:-Dump process command line gp_dump -p 8004 -U gpadmin --gp-d=db_dumps/20151028 --gp-r=/data/master/eddie_4351-1/db_dumps/20151028 --gp-s=p --gp-k=20151028113351 --no-lock --gp-c --no-expand-children fpl_test 20151028:11:33:58:006542 gpcrondump:mdw:gpadmin-[INFO]:-Starting Dump process 20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:-Dump process returned exit code 0 20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:-Timestamp key = 20151028113351 20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:-Checked master status file and master dump file. 20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:-Dump status report 20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:---------------------------------------------------- 20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:-Target database = fpl_test 20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:-Dump subdirectory = 20151028 20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:-Dump type = Full database 20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:-Clear old dump directories = Off 20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:-Dump start time = 11:33:51 20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:-Dump end time = 11:34:00 20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:-Status = COMPLETED 20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:-Dump key = 20151028113351 20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:-Dump file compression = On 20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:-Vacuum mode type = Off 20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:-Exit code zero, no warnings generated 20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:---------------------------------------------------- 20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[WARNING]:-Found neither /usr/local/GP-4.3.5.1/bin/mail_contacts nor /home/gpadmin/mail_contacts 20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[WARNING]:-Unable to send dump email notification 20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:-To enable email notification, create /usr/local/GP-4.3.5.1/bin/mail_contacts or /home/gpadmin/mail_contacts containing required email addresses
6. Create another test table
create table fpl.x5(a int); fpl_test=# \dt fpl.* List of relations Schema | Name | Type | Owner | Storage --------+------+-------+---------+--------- fpl | x1 | table | gpadmin | heap fpl | x2 | table | gpadmin | heap fpl | x3 | table | gpadmin | heap fpl | x4 | table | gpadmin | heap fpl | x5 | table | gpadmin | heap (5 rows) 7. Take incremental Backup to add x5 [gpadmin@mdw ~]$ gpcrondump --incremental -x fpl_test 20151028:11:38:15:008524 gpcrondump:mdw:gpadmin-[INFO]:-Starting gpcrondump with args: --incremental -x fpl_test 20151028:11:38:15:008524 gpcrondump:mdw:gpadmin-[INFO]:-Bypassing disk space checks for incremental backup 20151028:11:38:15:008524 gpcrondump:mdw:gpadmin-[INFO]:---------------------------------------------------- 20151028:11:38:15:008524 gpcrondump:mdw:gpadmin-[INFO]:-Master Greenplum Instance dump parameters 20151028:11:38:15:008524 gpcrondump:mdw:gpadmin-[INFO]:---------------------------------------------------- 20151028:11:38:15:008524 gpcrondump:mdw:gpadmin-[INFO]:-Dump type = Incremental 20151028:11:38:15:008524 gpcrondump:mdw:gpadmin-[INFO]:-Database to be dumped = fpl_test 20151028:11:38:15:008524 gpcrondump:mdw:gpadmin-[INFO]:-Master port = 8004 20151028:11:38:15:008524 gpcrondump:mdw:gpadmin-[INFO]:-Master data directory = /data/master/eddie_4351-1/ 20151028:11:38:15:008524 gpcrondump:mdw:gpadmin-[INFO]:-Run post dump program = Off 20151028:11:38:15:008524 gpcrondump:mdw:gpadmin-[INFO]:-Rollback dumps = Off 20151028:11:38:15:008524 gpcrondump:mdw:gpadmin-[INFO]:-Dump file compression = On 20151028:11:38:15:008524 gpcrondump:mdw:gpadmin-[INFO]:-Clear old dump files = Off 20151028:11:38:15:008524 gpcrondump:mdw:gpadmin-[INFO]:-Update history table = Off 20151028:11:38:15:008524 gpcrondump:mdw:gpadmin-[INFO]:-Secure config files = Off 20151028:11:38:15:008524 gpcrondump:mdw:gpadmin-[INFO]:-Dump global objects = Off 20151028:11:38:15:008524 gpcrondump:mdw:gpadmin-[INFO]:-Vacuum mode type = Off Continue with Greenplum dump Yy|Nn (default=N): > y 20151028:11:38:21:008524 gpcrondump:mdw:gpadmin-[INFO]:-Directory /data/master/eddie_4351-1/db_dumps/20151028 exists 20151028:11:38:21:008524 gpcrondump:mdw:gpadmin-[INFO]:-Checked /data/master/eddie_4351-1/ on master 20151028:11:38:22:008524 gpcrondump:mdw:gpadmin-[INFO]:-Adding compression parameter 20151028:11:38:22:008524 gpcrondump:mdw:gpadmin-[INFO]:-Adding --incremental 20151028:11:38:22:008524 gpcrondump:mdw:gpadmin-[INFO]:-Adding --no-expand-children 20151028:11:38:22:008524 gpcrondump:mdw:gpadmin-[INFO]:-Dump process command line gp_dump -p 8004 -U gpadmin --gp-d=db_dumps/20151028 --gp-r=/data/master/eddie_4351-1/db_dumps/20151028 --gp-s=p --gp-k=20151028113815 --no-lock --gp-c --incremental --no-expand-children fpl_test --table-file=/tmp/dirty_backup_list__SsEIq 20151028:11:38:22:008524 gpcrondump:mdw:gpadmin-[INFO]:-Starting Dump process 20151028:11:38:29:008524 gpcrondump:mdw:gpadmin-[INFO]:-Dump process returned exit code 0 20151028:11:38:29:008524 gpcrondump:mdw:gpadmin-[INFO]:-Timestamp key = 20151028113815 20151028:11:38:29:008524 gpcrondump:mdw:gpadmin-[INFO]:-Checked master status file and master dump file. 20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[INFO]:-Releasing pg_class lock 20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[INFO]:-Dump status report 20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[INFO]:---------------------------------------------------- 20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[INFO]:-Target database = fpl_test 20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[INFO]:-Dump subdirectory = 20151028 20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[INFO]:-Dump type = Incremental 20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[INFO]:-Clear old dump directories = Off 20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[INFO]:-Dump start time = 11:38:15 20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[INFO]:-Dump end time = 11:38:29 20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[INFO]:-Status = COMPLETED 20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[INFO]:-Dump key = 20151028113815 20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[INFO]:-Dump file compression = On 20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[INFO]:-Vacuum mode type = Off 20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[INFO]:-Exit code zero, no warnings generated 20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[INFO]:---------------------------------------------------- 20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[WARNING]:-Found neither /usr/local/GP-4.3.5.1/bin/mail_contacts nor /home/gpadmin/mail_contacts 20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[WARNING]:-Unable to send dump email notification 20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[INFO]:-To enable email notification, create /usr/local/GP-4.3.5.1/bin/mail_contacts or /home/gpadmin/mail_contacts containing required email addresses Backup completes successfully.
8. Drop the newly created table
fpl_test=# drop table fpl.x5; DROP TABLE fpl_test=# \dt fpl.*; List of relations Schema | Name | Type | Owner | Storage --------+------+-------+---------+--------- fpl | x1 | table | gpadmin | heap fpl | x2 | table | gpadmin | heap fpl | x3 | table | gpadmin | heap fpl | x4 | table | gpadmin | heap (4 rows)
9. Below are the contents of the file that needs to be restored, please note we have added the table to the list which we have already dropped:
fpl.x1 fpl.x2 fpl.x3 fpl.x4 fpl.x5
10. Try restoring using the incremental backup
[gpadmin@mdw ~]$ gpdbrestore --noanalyze -t 20151028113815 -a --truncate --table-file shubh/restore_dd 20151028:11:40:57:009817 gpdbrestore:mdw:gpadmin-[INFO]:-Starting gpdbrestore with args: --noanalyze -t 20151028113815 -a --truncate --table-file shubh/restore_dd 20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:------------------------------------------- 20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:-Greenplum database restore parameters 20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:------------------------------------------- 20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:-Restore type = Incremental Table Restore 20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:-Database name = fpl_test 20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:------------------------------------------- 20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:-Table restore list 20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:------------------------------------------- 20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:-Table = fpl.x1 20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:-Table = fpl.x2 20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:-Table = fpl.x3 20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:-Table = fpl.x4 20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:-Table = fpl.x5 20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:-Restore method = Specific table restore 20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:-Restore timestamp = 20151028113815 20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:-Restore compressed dump = On 20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:-Restore global objects = Off 20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:-Array fault tolerance = f 20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:------------------------------------------- 20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[CRITICAL]:-gpdbrestore failed. (Reason=''str' object is not callable') exiting...
Note- The restoration fails since the table is not present in the destination database, but it's name is present in the file list supplied.
The only solution is to create the extra table present in the file list at the destination database.
Feedback
Yes
No
Powered by
