Huge size of gpcc_exec* files generated in ccdata directory of GPCC causing directory full.

Products

• VMware Tanzu Greenplum

Issue/Introduction

Sometimes, we can see many large-sized gpcc_exec* files reside in ccdata directory of GPCC and the number of these files keeps increasing , which will put DB under potential risk of being down if the related directory is full.

In the example below, we can see GPCC directory is really big and gpcc_exec* files consume most of the space:

[root@sdw1 local]# du -sh greenplum-cc-6.9.1
180G greenplum-cc-6.9.1

[root@sdw1 greenplum-cc-6.9.1]# du -sh ccdata
180G ccdata

[root@sdw1 greenplum-cc-6.9.1]# cd ccdata
[root@sdw1 ccdata]# du -sh ./gpcc_exec*
12G gpcc_exec_0.dat
12G gpcc_exec_1.dat
12G gpcc_exec_2.dat
12G gpcc_exec_3.dat
12G gpcc_exec_4.dat
12G gpcc_exec_5.dat
12G gpcc_exec_6.dat
....
....

It is actually a known issue, which can be triggered by the fact that the PlanNodeHavester in webserver ends unexpectedly when gpcc agent once fails to start thus causing plannode data files being accumulated for days followed by disk bloating.

Environment

Product Version: 6.22

Resolution

The issue is expected to be fixed in GPCC 6.11. Current workaround is to monitor disk space on all the hosts, remove these gpcc_exec* files when they are reaching huge size and then restart GPCC.