GPCC "Warning: Updates stalled due to heavy GPDB workload or segment failure (system, diskusage)"

Products

• VMware Tanzu Greenplum

Issue/Introduction

GPCC WebUI displays:

Warning: Updates stalled due to heavy GPDB workload or segment failure (system)

Processes running on the master:

$ ps -aef | egrep 'con[0-9]'  | egrep 'seg-1|cmd'
gpadmin  31999 31961  0 12:47 ?        00:00:00 postgres:  5432, gpmon gpperfmon 192.168.50.51(47198) con9 cmd69 idle
gpadmin  32029 31961  0 12:47 ?        00:00:00 postgres:  5432, gpmon gpperfmon 192.168.50.51(47214) con11 cmd28 idle
gpadmin  32085 31961  0 12:48 ?        00:00:00 postgres:  5432, gpmon gpperfmon 192.168.50.51(47262) con9 seg-1 startup
gpadmin  32088 31961  0 12:48 ?        00:00:00 postgres:  5432, gpmon gpperfmon 192.168.50.51(47264) con9 seg-1 startup
gpadmin  32107 31961  0 12:48 ?        00:00:00 postgres:  5432, gpmon gpperfmon 192.168.50.51(47276) con9 seg-1 startup
gpadmin  32120 31961  0 12:48 ?        00:00:00 postgres:  5432, gpmon gpperfmon 192.168.50.51(47278) con9 seg-1 startup

Master log:

...,"LOG","00000","rejecting TCP connection to master using internalconnection protocol",,,,,,,0,,"auth.c",361,
...,"LOG","00000","An exception was encountered during the execution of statement: INSERT INTO gpmetrics.gpcc_pg_log_history SELECT * FROM gpmetrics._gpcc_pg_log_increment WHERE logseverity IN ('WARNING', 'ERROR', 'FATAL', 'PANIC');",,,,,,,0,,,,

If the database is started with the PGHOST or PGHOSTADDR is set when starting the database, then any Query Executors on the master will use the IP address or hostname to connect to the database and will not use the unix socket.

Environment

Product Version: 6.7

Resolution

Workaround

Set the GUC gp_reject_internal_tcp_connection=off

gpconfig -c gp_reject_internal_tcp_connection -v off --skipvalidation

Fix

When stopping and starting the database, ensure that neither PGHOST nor PGHOSTADDR is set. Remove these environment variables from the .bashrc and .bash_profile files if either are set on login.