gprecoverseg failed with : [Errno 2] No such file or directory:

Products

• VMware Tanzu Greenplum
• VMware Tanzu Greenplum / Gemfire

Issue/Introduction

When running gprecoverseg you encounter an error:

20200323:06:38:23:726847 gprecoverseg:cdw:gpadmin-[INFO]:-Starting gprecoverseg with args: -a
20200323:06:38:23:726847 gprecoverseg:cdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.24.0 build commit:114c7a6ff9e509b841c2afbed689895ec3a86c3c'
20200323:06:38:23:726847 gprecoverseg:cdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.24.0 build commit:114c7a6ff9e509b841c2afbed689895ec3a86c3c) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Dec 19 2019 05:50:16'
20200323:06:38:23:726847 gprecoverseg:cdw:gpadmin-[INFO]:-Checking if segments are ready to connect
20200323:06:38:23:726847 gprecoverseg:cdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20200323:06:38:26:726847 gprecoverseg:cdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20200323:06:38:29:726847 gprecoverseg:cdw:gpadmin-[INFO]:-Heap checksum setting is consistent between master and the segments that are candidates for recoverseg
20200323:06:38:29:726847 gprecoverseg:cdw:gpadmin-[INFO]:-Greenplum instance recovery parameters
20200323:06:38:29:726847 gprecoverseg:cdw:gpadmin-[INFO]:----------------------------------------------------------
20200323:06:38:29:726847 gprecoverseg:cdw:gpadmin-[INFO]:-Recovery type              = Standard
20200323:06:38:29:726847 gprecoverseg:cdw:gpadmin-[INFO]:----------------------------------------------------------
20200323:06:38:29:726847 gprecoverseg:cdw:gpadmin-[INFO]:-Recovery 1 of 16
20200323:06:38:29:726847 gprecoverseg:cdw:gpadmin-[INFO]:----------------------------------------------------------

(...)

20200323:11:38:44:135814 gprecoverseg:cdw:gpadmin-[INFO]:-16 segment(s) to recover
20200323:11:38:44:135814 gprecoverseg:cdw:gpadmin-[INFO]:-Ensuring 16 failed segment(s) are stopped
20200323:11:39:01:135814 gprecoverseg:cdw:gpadmin-[INFO]:-Ensuring that shared memory is cleaned up for stopped segments
20200323:11:39:02:135814 gprecoverseg:cdw:gpadmin-[INFO]:-Updating configuration with new mirrors
20200323:11:39:02:135814 gprecoverseg:cdw:gpadmin-[INFO]:-Updating mirrors
20200323:11:39:05:135814 gprecoverseg:cdw:gpadmin-[CRITICAL]:-Error occurred: Error Executing Command:
 Command was: 'ssh -o StrictHostKeyChecking=no -o ServerAliveInterval=60 sdw1 ". /usr/local/greenplum-db-5.24.0/greenplum_path.sh; $GPHOME/bin/lib/gpconfigurenewsegment -c \"/data/mirror/gpseg1:16734:false:false:35,/data/mirror/gpseg4:16735:false:false:38,/data/mirror/gpseg7:16736:false:false:41,/data/mirror/gpseg8:16737:false:false:42,/data/mirror/gpseg11:16738:false:false:45,/data/mirror/gpseg14:16739:false:false:48,/data/primary/gpseg16:16726:false:false:18,/data/primary/gpseg17:16727:false:false:19,/data/primary/gpseg18:16728:false:false:20,/data/primary/gpseg19:16729:false:false:21,/data/primary/gpseg20:16730:false:false:22,/data/primary/gpseg21:16731:false:false:23,/data/primary/gpseg22:16732:false:false:24,/data/primary/gpseg23:16733:false:false:25,/data/mirror/gpseg26:16740:
false:false:60,/data/mirror/gpseg29:16741:false:false:63\" -B 16 --write-gpid-file-only"'
rc=1, stdout='20200323:11:39:05:710729 gpconfigurenewsegment:sdw1:gpadmin-[INFO]:-Starting gpconfigurenewsegment with args: -c /data/mirror/gpseg1:16734:false:false:35,/data/mirror/gpseg4:16735:false:false:38,/data/mirror/gpseg7:16736:false:false:41,/data/mirror/gpseg8:16737:false:false:42,/data/mirror/gpseg11:16738:false:false:45,/data/mirror/gpseg14:16739:false:false:48,/data/primary/gpseg16:16726:false:false:18,/data/primary/gpseg17:16727:false:false:19,/data/primary/gpseg18:16728:false:false:20,/data/primary/gpseg19:16729:false:false:21,/data/primary/gpseg20:16730:false:false:22,/data/primary/gpseg21:16731:false:false:23,/data/primary/gpseg22:16732:false:false:24,/data/primary/gpseg23:16733:false:false:25,/data/mirror/gpseg26:16740:false:false:60,/data/mirror/gpseg29:16
741:false:false:63 -B 16 --write-gpid-file-only
20200323:11:39:05:710729 gpconfigurenewsegment:sdw1:gpadmin-[INFO]:-create gp_dbid file for segment
20200323:11:39:05:710729 gpconfigurenewsegment:sdw1:gpadmin-[INFO]:-create gp_dbid file for segment
20200323:11:39:05:710729 gpconfigurenewsegment:sdw1:gpadmin-[INFO]:-create gp_dbid file for segment
20200323:11:39:05:710729 gpconfigurenewsegment:sdw1:gpadmin-[INFO]:-create gp_dbid file for segment
20200323:11:39:05:710729 gpconfigurenewsegment:sdw1:gpadmin-[INFO]:-create gp_dbid file for segment
20200323:11:39:05:710729 gpconfigurenewsegment:sdw1:gpadmin-[INFO]:-create gp_dbid file for segment
20200323:11:39:05:710729 gpconfigurenewsegment:sdw1:gpadmin-[INFO]:-create gp_dbid file for segment
20200323:11:39:05:710729 gpconfigurenewsegment:sdw1:gpadmin-[INFO]:-create gp_dbid file for segment
20200323:11:39:05:710729 gpconfigurenewsegment:sdw1:gpadmin-[INFO]:-create gp_dbid file for segment
20200323:11:39:05:710729 gpconfigurenewsegment:sdw1:gpadmin-[INFO]:-create gp_dbid file for segment
20200323:11:39:05:710729 gpconfigurenewsegment:sdw1:gpadmin-[INFO]:-create gp_dbid file for segment
20200323:11:39:05:710729 gpconfigurenewsegment:sdw1:gpadmin-[INFO]:-create gp_dbid file for segment
20200323:11:39:05:710729 gpconfigurenewsegment:sdw1:gpadmin-[INFO]:-create gp_dbid file for segment
20200323:11:39:05:710729 gpconfigurenewsegment:sdw1:gpadmin-[INFO]:-create gp_dbid file for segment
20200323:11:39:05:710729 gpconfigurenewsegment:sdw1:gpadmin-[INFO]:-create gp_dbid file for segment
20200323:11:39:05:710729 gpconfigurenewsegment:sdw1:gpadmin-[INFO]:-create gp_dbid file for segment
20200323:11:39:05:710729 gpconfigurenewsegment:sdw1:gpadmin-[INFO]:-create gp_dbid file for segment
20200323:11:39:05:710729 gpconfigurenewsegment:sdw1:gpadmin-[INFO]:-create gp_dbid file for segment
20200323:11:39:05:710729 gpconfigurenewsegment:sdw1:gpadmin-[ERROR]:-ExecutionError: 'Error Executing Command: ' occured.  Details: '/usr/local/greenplum-db-5.24.0/bin/lib/gpconfigurenewsegment -c /data/mirror/gpseg1:16734:false:false:35,/data/mirror/gpseg4:16735:false:false:38,/data/mirror/gpseg7:16736:false:false:41,/data/mirror/gpseg8:16737:false:false:42,/data/mirror/gpseg11:16738:false:false:45,/data/mirror/gpseg14:16739:false:false:48,/data/primary/gpseg16:16726:false:false:18,/data/primary/gpseg17:16727:false:false:19,/data/primary/gpseg18:16728:false:false:20,/data/primary/gpseg19:16729:false:false:21,/data/primary/gpseg20:16730:false:false:22,/data/primary/gpseg21:16731:false:false:23,/data/primary/gpseg22:16732:false:false:24,/data/primary/gpseg23:16733:false:false:25,/data/mirror/gpseg26:16740:false:false:60,/data/mirror/gpseg29:16741:false:false:63 -B 16 --write-gpid-file-only'  cmd had rc=1 completed=True halted=False
  stdout=''
  stderr='[Errno 2] No such file or directory: '/data/mirror/gpseg1/gp_dbid''
'

Environment

Product Version: 5.24

Resolution

Check the error location if you can access the gp_dbid file
 

ssh sdw1
[gpadmin@sdw1_lab1 ~]$ ls -ltrh /data/mirror/gpseg1/gp_dbid 
ls: cannot access /data/mirror/gpseg1/gp_dbid: No such file or directory

If you get an error that the file does not exist check if you can see the directory

[gpadmin@sdw1_lab1 ~]$ ls -ltrh /data/mirror/gpseg1 
ls: cannot access /data/mirror/gpseg1: No such file or directory I

If you can't see a directory like in the example above, most likely this is related to a hardware issue. Verify, if this is a network mount or physical disk and ask the OS hardware admin to check the underlying storage for errors. Only if it is confirmed the underlying storage is ok and you can create a file on this location with touch you might continue to recovery

If this is the network mount that was mounted back and you can see the segment files you might need only to run a regular recovery.

If this is a hardware issue and the segment directory is gone permanently you need to run gprecoverseg -F to fully copy back the missing segment.