Segments not coming up online with gprecoverseg: "Failed to inform primary segment of updated mirroring state"

Products

• VMware Tanzu Greenplum

Issue/Introduction

You receive the following error when issuing gprecoverseg to recover failed segments:

20190520:16:01:22:032763 gprecoverseg:hostname_master:gpadmin2-[WARNING]:-Failed to inform primary segment of updated mirroring state. Segment: hostname_primary:/Data/greenplumdb/mirror/gpseg4:content=4:dbid=14:mode=r:status=u: REASON: Conversion failed. stdout:"" stderr:"failure: Error: MirroringFailure failure: Error: MirroringFailure "

Master pg_log shows the following:

2019-05-22 13:25:18.564038 CDT,,,p7521,th1114392320,,,,0,con2,,seg-1,,,,,"LOG","00000","FTS: segment (dbid=14, content=4) reported fault FaultMirror segmentstatus 11 to the prober.",,,,,,,0,,,,

On the primary pg_log:

2019-05-22 13:24:06.631526 CDT,,,p27482,th1647585152,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2019-05-22 13:24:37.207232 CDT,,,p21423,th1647585152,,,1999-12-31 18:00:00 CST,0,,,seg-1,,,,,"WARNING","XX000","receive close on connection: Success (cdbfilerepconnserver.c:337)",,,,,,,0,,"cdbfilerepconnserver.c",337,
2019-05-22 13:24:37.207297 CDT,,,p27482,th1647585152,,,,0,,,seg-1,,,,,"WARNING","01000","PostmasterPrimaryMirrorTransition (3) Finished with Error",,,,,,,0,,"primary_mirror_mode.c",1707,

When doing a strace on the primary sender process, you can see that the following connection was not successful:

20162 16:27:38.828541 connect(5<TCP:[79300821]>, {sa_family=AF_INET, sin_port=htons(46000), sin_addr=inet_addr("192.168.30.1")}, 16) = -1 EINPROGRESS (Operation now in progress) <0.000078>
20162 16:27:38.829597 poll([{fd=5<TCP:[192.168.30.2:24874->192.168.30.1:46000]>, events=POLLOUT|POLLERR}], 1, 600000) = 1 ([{fd=5, revents=POLLOUT|POLLERR|POLLHUP}]) <1.006675>
20162 16:27:39.837333 getsockopt(5<TCP:[192.168.30.2:24874->192.168.30.1:46000]>, SOL_SOCKET, SO_ERROR, [113], [4]) = 0 <0.000057>
20162 16:27:39.837537 close(5<TCP:[192.168.30.2:24874->192.168.30.1:46000]>) = 0 <0.000092>
20162 16:27:39.837737 kill(27482, SIG_0) = 0 <0.000034>

Environment

Product Version: 4.3

Resolution

There is most likely a connectivity issue between primary and mirror segment hosts.

Check if you can connect from the primary segment host to the mirror host on the replication port:

[gpadmin2@hostname_primary]$ nc -v -z 192.168.30.1 46000
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: No route to host.

Fix the connectivity issue - reboot the segment hosts if necessary, or contact your OS/networking admin.