Greenplum utilities report error message "stderr='ssh_exchange_identification: read: Connection reset by peer" or "kex_exchange_identification: Connection closed by remote host""

Products

• VMware Tanzu Greenplum

Issue/Introduction

Greenplum utilities report error like:

ssh_exchange_identification: read: Connection reset by peer

or

kex_exchange_identification: Connection closed by remote host

For example:

[gpadmin@mdw pg_log]$ gpstop -u
20190227:14:50:24:275338 gpstop:cdw:gpadmin-[INFO]:-Starting gpstop with args: -u
20190227:14:50:24:275338 gpstop:cdw:gpadmin-[INFO]:-Gathering information and validating the environment...
20190227:14:50:24:275338 gpstop:cdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
20190227:14:50:24:275338 gpstop:cdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20190227:14:50:26:275338 gpstop:cdw:gpadmin-[INFO]:-Greenplum Version: 'postgres (Greenplum Database) 6.25.4 build 1'
20190227:14:50:26:275338 gpstop:cdw:gpadmin-[INFO]:-Signalling all postmaster processes to reload
   :
   :
20190227:14:50:31:275338 gpstop:cdw:gpadmin-[CRITICAL]:-Error occurred: Error Executing Command: 
Command was: 'ssh -o StrictHostKeyChecking=no -o ServerAliveInterval=60 sdw3 ". /usr/local/greenplum-db/./greenplum_path.sh; $GPHOME/bin/pg_ctl reload -D /data2/primary/gpseg20"'
rc=255, stdout='', stderr='ssh_exchange_identification: read: Connection reset by peer' 

 

Cause

The option of MaxStartups 10:30:300 in /etc/ssh/sshd.conf means that sshd could refuse the 30% of unauthenticated connections according to the sshd_conf man page as below.

MaxStartups

Specifies the maximum number of concurrent unauthenticated connections to the SSH daemon.  Additional connections will be dropped until authentication succeeds or the LoginGraceTime expires for a connection.

The default is 10:30:100.

Alternatively, random early drop can be enabled by specifying the three colon separated values start:rate:full (e.g. "10:30:60").  sshd(8) will refuse connection attempts with a probability of rate/100 (30%) if there are currently start (10) unauthenticated connections. The probability increases linearly and all connection attempts are refused if the number of unauthenticated connections reaches full (60).

In a Greenplum environment the coordinator host can start a large number of ssh connections to a segment host. If the segment host is not able to authenticate and complete the connections quickly enough, a number of the connections may be denied due to the above explaination of MaxStartups setting.
 

Resolution

Alter the MaxStartups setting  in /etc/ssh/sshd_config on all hosts in the cluster to allow a sufficient number of unauthenticated connections. 

Note: root user access is required for these changes

​$ vi /etc/ssh/sshd_config
~~ skip
MaxStartups 50:5:300
​~~ skip
 
$ systemctl restart sshd # restart the sshd service for the changes to take effect