gprecoverseg fails with error 'could not open target file "/storage/nssd7/data/primary/gpseg114/gpbackup_history.yaml"'
Article ID: 296811
Updated On:
Products
VMware Tanzu Greenplum
Issue/Introduction
gprecoverseg output is similar to :
: : 20220511:03:19:12:124074 gprecoverseg:ddlgpmdev11a:gpadmin-[INFO]:----------------------------------------------------------- 20220511:03:19:12:124074 gprecoverseg:ddlgpmdev11a:gpadmin-[INFO]:-Failed to recover the following segments. You must run gprecoverseg -F for all incremental failures 20220511:03:19:12:124074 gprecoverseg:ddlgpmdev11a:gpadmin-[INFO]:- hostname: sdw10; port: 6006; logfile: /home/gpadmin/gpAdminLogs/pg_rewind.20220511_031848.dbid188.out; recoverytype: incremental 20220511:03:19:12:124074 gprecoverseg:ddlgpmdev11a:gpadmin-[INFO]:- hostname: sdw10; port: 6010; logfile: /home/gpadmin/gpAdminLogs/pg_rewind.20220511_031848.dbid192.out; recoverytype: incremental 20220511:03:19:12:124074 gprecoverseg:ddlgpmdev11a:gpadmin-[INFO]:- hostname: sdw10; port: 6009; logfile: /home/gpadmin/gpAdminLogs/pg_rewind.20220511_031848.dbid191.out; recoverytype: incremental 20220511:03:19:12:124074 gprecoverseg:ddlgpmdev11a:gpadmin-[INFO]:- hostname: sdw10; port: 6004; logfile: /home/gpadmin/gpAdminLogs/pg_rewind.20220511_031848.dbid186.out; recoverytype: incremental 20220511:03:19:12:124074 gprecoverseg:ddlgpmdev11a:gpadmin-[INFO]:- hostname: sdw10; port: 6008; logfile: /home/gpadmin/gpAdminLogs/pg_rewind.20220511_031848.dbid190.out; recoverytype: incremental 20220511:03:19:12:124074 gprecoverseg:ddlgpmdev11a:gpadmin-[INFO]:- hostname: sdw10; port: 6007; logfile: /home/gpadmin/gpAdminLogs/pg_rewind.20220511_031848.dbid189.out; recoverytype: incremental 20220511:03:19:12:124074 gprecoverseg:ddlgpmdev11a:gpadmin-[INFO]:- hostname: sdw10; port: 6003; logfile: /home/gpadmin/gpAdminLogs/pg_rewind.20220511_031848.dbid185.out; recoverytype: incremental 20220511:03:19:12:124074 gprecoverseg:ddlgpmdev11a:gpadmin-[INFO]:- hostname: sdw10; port: 6001; logfile: /home/gpadmin/gpAdminLogs/pg_rewind.20220511_031848.dbid183.out; recoverytype: incremental 20220511:03:19:12:124074 gprecoverseg:ddlgpmdev11a:gpadmin-[INFO]:- hostname: sdw10; port: 6002; logfile: /home/gpadmin/gpAdminLogs/pg_rewind.20220511_031848.dbid184.out; recoverytype: incremental 20220511:03:19:12:124074 gprecoverseg:ddlgpmdev11a:gpadmin-[INFO]:- hostname: sdw10; port: 6000; logfile: /home/gpadmin/gpAdminLogs/pg_rewind.20220511_031848.dbid182.out; recoverytype: incremental 20220511:03:19:12:124074 gprecoverseg:ddlgpmdev11a:gpadmin-[INFO]:- hostname: sdw10; port: 6011; logfile: /home/gpadmin/gpAdminLogs/pg_rewind.20220511_031848.dbid193.out; recoverytype: incremental 20220511:03:19:12:124074 gprecoverseg:ddlgpmdev11a:gpadmin-[INFO]:- hostname: sdw10; port: 6005; logfile: /home/gpadmin/gpAdminLogs/pg_rewind.20220511_031848.dbid187.out; recoverytype: incremental 20220511:03:19:12:124074 gprecoverseg:ddlgpmdev11a:gpadmin-[INFO]:-Triggering FTS probe 20220511:03:19:12:124074 gprecoverseg:ddlgpmdev11a:gpadmin-[ERROR]:-gprecoverseg failed. Please check the output for more details.The pg_rewind log files specified in the above output show:
$ cat /home/gpadmin/gpAdminLogs/pg_rewind.20220511_031848.dbid188.out; connected to server 2022-05-11 03:18:49.159273 EDT,,,p22787,th-1210247040,,,,0,,,seg114,,,,,"LOG","00000","gpdb dir for cgroup component ""cpu"": /sys/fs/cgroup/cpu/gpdb/",,,,,,,,"dumpCompDirs","resgroup-ops-linux.c",472, 2022-05-11 03:18:49.159701 EDT,,,p22787,th-1210247040,,,,0,,,seg114,,,,,"LOG","00000","gpdb dir for cgroup component ""cpuacct"": /sys/fs/cgroup/cpuacct/gpdb/",,,,,,,,"dumpCompDirs","resgroup-ops-linux.c",472, 2022-05-11 03:18:49.160071 EDT,,,p22787,th-1210247040,,,,0,,,seg114,,,,,"LOG","00000","gpdb dir for cgroup component ""memory"": /sys/fs/cgroup/memory/gpdb/",,,,,,,,"dumpCompDirs","resgroup-ops-linux.c",472, 2022-05-11 03:18:49.160446 EDT,,,p22787,th-1210247040,,,,0,,,seg114,,,,,"LOG","00000","gpdb dir for cgroup component ""cpuset"": /sys/fs/cgroup/cpuset/gpdb/",,,,,,,,"dumpCompDirs","resgroup-ops-linux.c",472, servers diverged at WAL position 26F8/DFC96CC0 on timeline 3 rewinding from last common checkpoint at 26F8/CAE43428 on timeline 3 reading source file list reading target file list reading WAL in target need to copy 2996 MB (total source directory size is 2129793 MB) could not open target file "/storage/nssd7/data/primary/gpseg114/gpbackup_history.yaml": Permission denied Failure, exiting
Environment
Product Version: 6.20
Resolution
Run "ls -l" on the file in the primary and mirror directory of the segment.
If the permissions do not allow the "gpadmin" user to write to the file then the recovery will fail.
In the example above the file could be deleted from the primary and mirror segment directories as it was not needed.
The "gpbackup_history.yaml" file should only be on the master host.
Change the permissions and/or ownership of the file in the primary and mirror directories to allow the user "gpadmin" to write to the file and retry the gprecoverseg.
If the permissions do not allow the "gpadmin" user to write to the file then the recovery will fail.
In the example above the file could be deleted from the primary and mirror segment directories as it was not needed.
The "gpbackup_history.yaml" file should only be on the master host.
Change the permissions and/or ownership of the file in the primary and mirror directories to allow the user "gpadmin" to write to the file and retry the gprecoverseg.
Feedback
Yes
No
Powered by
