pg_terminate_backend() left one process alive on one segment.

Products

• VMware Tanzu Data Suite
• VMware Tanzu Greenplum
• Greenplum

Issue/Introduction

pg_terminate_backend() failed to kill all the segment processes, One left on one of the segments.

[[e-mail] ~]$ gpssh -f hostfile 
Running gpssh
>>> ps -ef |grep -i con44651


gpadmin  297930  19811  0 03:28 ?        00:00:01 postgres: 40008, insert_into_table1 prod 10.1010.10(53678) con44651 seg80 cmd298 MPPEXEC INSERT
total segment process found = 1

Cause

The cause for this exact issue is unknown as we did not have all the artifacts to RCA. 

Resolution

Workaround :

 Use utility mode psql to login the segment's postmaster, and select pg_terminate_backend(pid);

The R&D team are investigating a code refine to help in such cases: 

We are refining the code so that if the QD(master process) is gone, QEs(server processes) can detect and automatically exit so we  will not be left with un-exited QEs