Greenplum 6.x: GPCC causing segment crashes

Products

• VMware Tanzu Greenplum

Issue/Introduction

In Greenplum 6.0 - 6.7.1, you may face the following issue if you are using GPCC 4.x and above:

2020-09-18 12:34:04.253914 EDT,,,p304924,th0,,,2020-09-18 12:26:44 EDT,0,con504392,cmd240,seg18,,,,,"PANIC","XX000","Unexpected internal error: Segment process received signal SIGSEGV",,,,,,,0,,,,"1    0x7f213debe630 libpthread.so.0 <symbol not found> + 0x3debe630
2    0x829eac postgres <symbol not found> (copy.c:5456)
3    0x82fd94 postgres <symbol not found> (copy.c:3904)
4    0x833e30 postgres DoCopy (copy.c:1129)
5    0xa87105 postgres standard_ProcessUtility (utility.c:637)
6    0xa83711 postgres <symbol not found> (palloc.h:158)
7    0xa8420e postgres <symbol not found> (pquery.c:1512)
8    0xa85927 postgres PortalRun (pquery.c:1022)
9    0xa7de1e postgres <symbol not found> (postgres.c:1377)
10   0xa82a58 postgres PostgresMain (postgres.c:5403)
"

In the master or segment log, you can try to trace which query caused the crash by searching for con<session_id>. In this case it is con504392. If you see a query in this form that is run by gpsmon user, then it is caused by GPCC:

COPY gpmetrics.gpcc_queries_history FROM '/usr/local/greenplum-db/greenplum-cc-web-6.1.0/ccdata/gpcc_queryinfo_2020-09-18_123403.dat' ESCAPE E'\\\\' CSV NULL AS E'null' DELIMITER E'|' LOG ERRORS SEGMENT REJECT LIMIT 100 PERCENT;

In general, this issue applies to all COPY statements, but GPCC runs these kinds of statements regularly as part of background data collection.

Environment

Product Version: 6.7

Resolution

This issue is fixed in Greenplum 6.8.0. If you see this issue, please upgrade to this version or above and in the meantime you can stop GPCC. If the query was a generic COPY statement, then refrain from running it until you upgrade.

Release Notes:
172576000 - COPY
If data format errors occurred while copying data into a partitioned table with a COPY FROM command in single row error isolation mode, Greenplum Database might crash when a query executor (QE) did not handle the data format error correctly. This issue is resolved.