"Host key verification failed" when running gpexpand

Products

• VMware Tanzu Greenplum
• VMware Tanzu Greenplum / Gemfire

Issue/Introduction

gpexpand fails with the below error:

gpadmin@cdw $ gpexpand -i gpexpand_new
20201211:23:46:34:908345 gpexpand:cdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 6.12.0 build commit:4c176763c7619fb678ce38095e6b3e8fb9548186'
20201211:23:46:34:908345 gpexpand:cdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 9.4.24 (Greenplum Database 6.12.0 build commit:4c176763c7619fb678ce38095e6b3e8fb9548186) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Oct 28 2020 19:42:15' 
20201211:23:46:34:908345 gpexpand:cdw:gpadmin-[INFO]:-Querying gpexpand schema for current expansion state 
20201211:23:46:37:908345 gpexpand:cdw:gpadmin-[WARNING]:-cannot access pg_controldata for dbid 59 on host sdw4 
20201211:23:46:37:908345 gpexpand:cdw:gpadmin-[WARNING]:-cannot access pg_controldata for dbid 67 on host sdw3 
20201211:23:46:37:908345 gpexpand:cdw:gpadmin-[WARNING]:-cannot access pg_controldata for dbid 31 on host sdw1 
20201211:23:46:37:908345 gpexpand:cdw:gpadmin-[WARNING]:-cannot access pg_controldata for dbid 32 on host sdw1 
20201211:23:46:37:908345 gpexpand:cdw:gpadmin-[WARNING]:-cannot access pg_controldata for dbid 75 on host sdw2 
20201211:23:46:37:908345 gpexpand:cdw:gpadmin-[WARNING]:-cannot access pg_controldata for dbid 37 on host sdw5 
20201211:23:46:37:908345 gpexpand:cdw:gpadmin-[WARNING]:-cannot access pg_controldata for dbid 39 on host sdw5 
20201211:23:46:37:908345 gpexpand:cdw:gpadmin-[WARNING]:-cannot access pg_controldata for dbid 38 on host sdw5 
20201211:23:46:37:908345 gpexpand:cdw:gpadmin-[WARNING]:-cannot access pg_controldata for dbid 79 on host sdw2 
20201211:23:46:37:908345 gpexpand:cdw:gpadmin-[WARNING]:-cannot access pg_controldata for dbid 80 on host sdw3 
20201211:23:46:37:908345 gpexpand:cdw:gpadmin-[WARNING]:-cannot access pg_controldata for dbid 81 on host sdw4 
20201211:23:46:37:908345 gpexpand:cdw:gpadmin-[INFO]:-Heap checksum setting consistent across cluster 
20201211:23:46:37:908345 gpexpand:cdw:gpadmin-[INFO]:-Syncing Greenplum Database extensions 
20201211:23:46:38:908345 gpexpand:cdw:gpadmin-[INFO]:-The packages on sdw6 are consistent. 
20201211:23:46:39:908345 gpexpand:cdw:gpadmin-[INFO]:-The packages on sdw7 are consistent. 
20201211:23:46:39:908345 gpexpand:cdw:gpadmin-[INFO]:-Locking catalog 
20201211:23:46:39:908345 gpexpand:cdw:gpadmin-[INFO]:-Locked catalog 
20201211:23:46:39:908345 gpexpand:cdw:gpadmin-[INFO]:-Creating segment template 
20201211:23:46:41:908345 gpexpand:cdw:gpadmin-[INFO]:-Copying postgresql.conf from existing segment into template 
20201211:23:46:42:908345 gpexpand:cdw:gpadmin-[ERROR]:-gpexpand failed: ExecutionError: 'non-zero rc: 1' occurred. 
Details: 'ssh -o StrictHostKeyChecking=no -o ServerAliveInterval=60 sdw4 ". /usr/local/greenplum-db-6.12.0/greenplum_path.sh; /bin/scp /data1/primary/gpseg0/postgresql.conf cdw:/data/master/gpexpand_12112020_908345"' cmd had rc=1 completed=True halted=False 
stdout='' 
stderr=' Host key verification failed. lost connection ' 
Exiting... 
20201211:23:46:42:908345 gpexpand:cdw:gpadmin-[ERROR]:-Please run 'gpexpand -r' to rollback to the original state. 
20201211:23:46:42:908345 gpexpand:cdw:gpadmin-[INFO]:-Shutting down gpexpand...

 

Environment

Greenplum Database

Resolution

Verify that it is possible to ssh to all the old and new segments hosts without a password.

Use gpssh-keyex to configure passwordless ssh connections between the hosts.

See Enabling Passwordless SSH for more details.

Once passwordless SSH is congifured, rerun the gpexpand.