GPRestore failing due to broken pipe error after seeing "Expected to restore ## rows to table <table>, but restored ## instead"
search cancel

GPRestore failing due to broken pipe error after seeing "Expected to restore ## rows to table <table>, but restored ## instead"

book

Article ID: 297384

calendar_today

Updated On:

Products

VMware Tanzu Greenplum VMware Tanzu Greenplum / Gemfire VMware Tanzu Data Suite VMware Tanzu Data Suite

Issue/Introduction

Customer is attempting to restore a backup with GPRestore but seeing the number of restored rows in the table is incomplete compared to how many there are.  This causes the backup to fail.

This problem can exist if the customer is using the --single-data-file argument with the GPBackup.

An example of the error that may be seen: 
[ERROR]:-Expected to restore 50 rows to table <table name>, but restored 47 instead

Additionally, the GPBackup log itself may show the following: 
2022-03-07 16:11:09.151550 CET,"gpadmin","EXAMPLE_DATABASE",p295893,th-838342784,"11.2.3.5","55534",2022-03-07 15:51:06 CET,0,con420103,cmd61717,seg38,,dx4677705,,sx1,"LOG","00000","read err msg from pipe, len:106 msg:cat: /data/primary/gpseg38/gpbackup_38_20220305193001_pipe_65476_2215215: No such file or directory

In the GPBackup Helper logs on the segment that saw the error (--verbose logging argument may be required) the below message may be seen: 
/data/primary/gpseg18/gpbackup_18_20220305193001_pipe_223313_2215236
20220306:12:18:21 gpbackup_helper:gpadmin:sdw1:336227-[ERROR]:-Segment 18: 12:18:16 INFO  ddboost_plugin.c:1333: Initializing Data Domain connections
12:18:17 INFO  ddboost_plugin.c:1963: Subset file read from DataDomain file GPDB/20220305/20220305193001/gpbackup_18_20220305193001
: write /data/primary/gpseg18/gpbackup_18_20220305193001_pipe_223313_2215236: broken pipe: goroutine 1 [running]:
runtime/debug.Stack(0x950dc0, 0xc001039580, 0xc000226840)
        /usr/local/go/src/runtime/debug/stack.go:24 +0x9d
github.com/greenplum-db/gpbackup/helper.DoHelper()
        /tmp/build/3e49593f/go/src/github.com/greenplum-db/gpbackup/helper/helper.go:90 +0x164
main.main()
        /tmp/build/3e49593f/go/src/github.com/greenplum-db/gpbackup/gpbackup_helper.go:10 +0x20


Environment

 

Resolution

There are multiple potential causes of this particular issue.  Here are some (but not all) possibilities to consider:

  • Security software (e.g. CrowdStrike Falcon) have been observed to cause these GPRestore failures 
    • If the customer is willing, temporarily disable the security software to confirm
  • Disk and/or file system issues preventing read/write on the file system
    • Check 'dmesg -T' or messages log to confirm this
  • Scripts or other applications that could be interfering with IO
    • Check with the customer whether any changes to the applications have been made before the restores started to fail


Powered by Wolken Software