gpstop fails with "kex_exchange_identification: Connection closed by remote host"

Products

• VMware Tanzu Greenplum

Issue/Introduction

gpstop fails with the following error:

gpadmin@mdw $ gpstop -u
20210302:12:59:01:236566 gpstop:mdw:gpadmin-[INFO]:-Starting gpstop with args: -u
20210302:12:59:01:236566 gpstop:mdw:gpadmin-[INFO]:-Gathering information and validating the environment...
20210302:12:59:01:236566 gpstop:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
20210302:12:59:01:236566 gpstop:mdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20210302:12:59:02:236566 gpstop:mdw:gpadmin-[INFO]:-Greenplum Version: 'postgres (Greenplum Database) 4.3.29.0 build 1'
20210302:12:59:02:236566 gpstop:mdw:gpadmin-[INFO]:-Signalling all postmaster processes to reload
.
20210302:12:59:03:236566 gpstop:mdw:gpadmin-[CRITICAL]:-Error occurred: Error Executing Command:
Command was: 'ssh -o StrictHostKeyChecking=no -o ServerAliveInterval=60 sdw1 ". /usr/local/greenplum-db/./greenplum_path.sh; $GPHOME/bin/pg_ctl reload -D /data1/mirror/gpseg18"'
rc=255, stdout='', stderr='/etc/opt/adnssh/ssh_config line 39: Unsupported option "rsaauthentication"
/etc/opt/adnssh/ssh_config line 54: Unsupported option "rhostsrsaauthentication"
kex_exchange_identification: Connection closed by remote host
'

Environment

Product Version: 4.3.29

Resolution

This issue is caused by the sshd configuration file that has AuthorizedKeysFile configured:

cat /etc/ssh/sshd_config
(...)
AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2 /etc/ssh/authorized_keys.d/%u
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
(...)

gpstop starts up to 64 connections to all segments to communicate with them by default. ssh daemon is not able to load many authorization keys at the same time because they are not served fast enough. As a result. some of the connections are failing with  Connection closed by remote host .

Workaround

Decrease the number of connections used by the greenplum utilities by using the -B option that controls the amount of parallel processes. In the most extreme situation, you can use gpstop -u -B 1 to communicate with only one segment at that time but gpstop/gpstart will take more time. In this case it was enough to use gpstop -u -B 12

Permanent solution

Authorization OS admin has to work on the RSA key server to be able to serve the keys faster on parallel request.