A cancelled query does not stop on one segment. The process on the master is also still alive.
Running “kill -15” or "kill -2" on the segment process did not allow it to quit.
Connecting directly to the segment and running the pg_cancel_backend and pg_terminate_backend did not clear the process on the segment.
The process continues to run and frequently logs Process interrupt for 'query cancel pending'
gpdb-2023-05-01_023841.csv:2023-05-01 03:32:12.086610 KSTS,"xxx_xxxx","database_name",p51101,th-783619968,"xx.xx.x.xxx","23608",2023-05-01 03:31:13 EDT,0,con1443563,,seg813,,,,,"LOG","00000","Process interrupt for 'query cancel pending' (postgres.c:5274)",,,,,,,0,,"postgres.c",3885,
The cause of this issue is that malloc() is not a function that can be safely called from a signal handler. It's not an async-signal-safe function.
The process was interrupted by SIGUSR1 during the malloc process, and then another call to malloc in the signal handler caused a deadlock and did not allow the process to complete and quit.
The issue is fixed in Greenplum 6.26.1 and higher versions.
See issues "32867" and "33128" in the Release Notes
32867 Resolved an issue where query cancellation was failing due to malloc calls getting deadlocked.
33128 Resolves an issue where a long running query was successfully cancelled on all segments except for one segment, due to an underlying deadlock issue.