gpinitstandby is hanging on "pg_hba.conf files updated successfully" in Greenplum

Products

VMware Tanzu Greenplum

Issue/Introduction

You issued a gpinitstandby to add standby to the configuration. However, the command is hanging:

[gpadmin@mdw ~]$ gpinitstandby -s smdw
20200709:11:12:46:098898 gpinitstandby:mdw:gpadmin-[INFO]:-Validating environment and parameters for standby initialization...
20200709:11:12:46:098898 gpinitstandby:mdw:gpadmin-[INFO]:-Checking for filespace directory /data/master/gpseg-1 on smdw
20200709:11:12:46:098898 gpinitstandby:mdw:gpadmin-[INFO]:------------------------------------------------------
20200709:11:12:46:098898 gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum standby master initialization parameters
20200709:11:12:46:098898 gpinitstandby:mdw:gpadmin-[INFO]:------------------------------------------------------
20200709:11:12:46:098898 gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum master hostname               = mdw.gphd.local
20200709:11:12:46:098898 gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum master data directory         = /data/master/gpseg-1
20200709:11:12:46:098898 gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum master port                   = 5432
20200709:11:12:46:098898 gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum standby master hostname       = smdw
20200709:11:12:46:098898 gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum standby master port           = 5432
20200709:11:12:46:098898 gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum standby master data directory = /data/master/gpseg-1
20200709:11:12:46:098898 gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum update system catalog         = On
20200709:11:12:46:098898 gpinitstandby:mdw:gpadmin-[INFO]:------------------------------------------------------
20200709:11:12:46:098898 gpinitstandby:mdw:gpadmin-[INFO]:- Filespace locations
20200709:11:12:46:098898 gpinitstandby:mdw:gpadmin-[INFO]:------------------------------------------------------
20200709:11:12:46:098898 gpinitstandby:mdw:gpadmin-[INFO]:-pg_system -> /data/master/gpseg-1
Do you want to continue with standby master initialization? Yy|Nn (default=N):
> y
20200709:11:13:07:098898 gpinitstandby:mdw:gpadmin-[INFO]:-Syncing Greenplum Database extensions to standby
20200709:11:13:07:098898 gpinitstandby:mdw:gpadmin-[INFO]:-The packages on smdw are consistent.
20200709:11:13:07:098898 gpinitstandby:mdw:gpadmin-[INFO]:-Adding standby master to catalog...
20200709:11:13:07:098898 gpinitstandby:mdw:gpadmin-[INFO]:-Database catalog updated successfully.
20200709:11:13:09:098898 gpinitstandby:mdw:gpadmin-[INFO]:-Updating pg_hba.conf file...
20200709:11:13:15:098898 gpinitstandby:mdw:gpadmin-[INFO]:-pg_hba.conf files updated successfully.

Environment

Product Version: Other

Resolution

Eventually the command completes, but in this case, it took 19 hours.

The reason it took this long is because the size of db_analyze directory in the $MASTER_DATA_DIRECTORY.

[gpadmin@smdw gpseg-1]$ du -sh ./db_analyze/
249G    ./db_analyze/

When you check the directory creation date on stadby host, observe that creation started at July 9th and finished at July 10th:

drwx------     3 gpadmin gpadmin   21 Jul  9 12:03 ..
drwx------     2 gpadmin gpadmin 4.0K Jul  9 12:03 20171022044152
drwx------     2 gpadmin gpadmin 4.0K Jul  9 12:03 20171022044450
(...)
drwx------     2 gpadmin gpadmin 4.0K Jul 10 05:21 20200708011631
drwx------ 40344 gpadmin gpadmin 1.3M Jul 10 05:21 .
drwx------     2 gpadmin gpadmin 4.0K Jul 10 05:21 20200708013610

The reason for this length in time is the massive db_analyze folder that was never cleared.

More information can be found in the following analyzedb documentation:

https://docs.vmware.com/en/VMware-Greenplum/7/greenplum-database/utility_guide-ref-analyzedb.html - The utility does not automatically remove old snapshot information. Over time, the snapshots can consume a large amount of disk space. To recover disk space, you can keep the most recent snapshot information and remove the older snapshots. You can also specify the --clean_last or --clean_all option to remove state files generated by analyzedb.

Workaround

To resolve this issue, run analyzedb with --clean_last.