gprecoverseg failed with error "failed to receive response: Connection reset by peer (errno: 104)"

Products

• VMware Tanzu Greenplum

Issue/Introduction

Symptoms:
gprecoverseg fails with the error "failed to receive response: Connection reset by peer (errno: 104)" as shown below: 

20181112:03:13:23:029993 gprecoverseg:hostname:gpadmin-[WARNING]:-Failed to start segment. The fault prober will shortly mark it as down. Segment: hostname:/data02/primary/gpseg17:content=17:dbid=19:mode=r:status=d: REASON: Start failed; check segment logfile. "failed to receive response: Connection res et by peer (errno: 104) Retrying no 1 failed to receive response: Connection reset by peer (errno: 104) Retrying no 2 failed to receive response: Connection reset by peer (errno: 104) Retrying no 3 failed to receive response: Connection reset by peer (errno: 104) Retrying no 4 failed to receive response: Connection reset by peer (errno: 104) Retrying no 5 failed to receive response: Connection reset by peer (errno: 104) Retrying no 6 failed to receive response: Connection reset by pee

Environment

Cause

The port for the mirror instance has been used by another OS process.

netstat -anp |grep 40001
tcp        0      0 0.0.0.0:40001           0.0.0.0:*               LISTEN      20619/rpc.statd
[root@hostname ~]# ps -ef |grep 20619
rpcuser   20619      1  0 Nov11 ?        00:00:00 /usr/sbin/rpc.statd
root      44162  44119  0 03:09 pts/0    00:00:00 grep --color=auto 20619

Resolution

After stop the Remote Procedure Call (RPC) process, gprecoveresg will complete successfully.