pg_terminate_backend() failed to kill all the segment processes, One left on one of the segments.
[[email protected] ~]$ gpssh -f hostfile
Running gpssh
>>> ps -ef |grep -i con44651
gpadmin 297930 19811 0 03:28 ? 00:00:01 postgres: 40008, insert_into_table1 prod 10.1010.10(53678) con44651 seg80 cmd298 MPPEXEC INSERT
total segment process found = 1
The cause for this exact issue is unknown as we did not have all the artifacts to RCA.
Workaround :
Use utility mode psql to login the segment's postmaster, and select pg_terminate_backend(pid);
The R&D team are investigating a code refine to help in such cases:
We are refining the code so that if the QD(master process) is gone, QEs(server processes) can detect and automatically exit so we will not be left with un-exited QEs