GPCC agent logs grows to massive size

Products

• VMware Tanzu Greenplum

Issue/Introduction

The GPCC log file grew to a massive size:

-rw-r--r-- 1 gpadmin gpadmin 139G Nov 19 10:39 /usr/local/greenplum-cc-web-4.8.0/logs/agent.log

Environment

Product Version: 5.21

Resolution

Files starts as expected:

2019-10-10 13:22:47 [INFO]connect to rpc server ddlgpmtst11a.us.dell.com:8899
2019-10-10 13:22:47 [INFO]ControlCommand: connection estabilshed ddlgpmtst11a.us.dell.com:8899
2019-10-10 13:22:47 [INFO]agent begin to report health status ddlgpstst001.us.dell.com 4d65822107fcfd52
2019-10-10 13:27:47 [INFO]Throughput profile 0msg/s, 0KB/s, #BATCH:1, #FAIL:0, #RETRY:0
2019-10-10 13:32:47 [INFO]Throughput profile 0msg/s, 0KB/s, #BATCH:0, #FAIL:0, #RETRY:0
2019-10-10 13:37:47 [INFO]Throughput profile 0msg/s, 0KB/s, #BATCH:1, #FAIL:0, #RETRY:0

But finishes with streaming error:

2019-11-19 11:14:11 [ERROR]Accept error: accept unixpacket /tmp/.s.GPMC.sock: accept4: too many open files
2019-11-19 11:14:11 [ERROR]Accept error: accept unixpacket /tmp/.s.GPMC.sock: accept4: too many open files
2019-11-19 11:14:11 [ERROR]Accept error: accept unixpacket /tmp/.s.GPMC.sock: accept4: too many open files
2019-11-19 11:14:11 [ERROR]Accept error: accept unixpacket /tmp/.s.GPMC.sock: accept4: too many open files
2019-11-19 11:14:11 [ERROR]Accept error: accept unixpacket /tmp/.s.GPMC.sock: accept4: too many open files
2019-11-19 11:14:11 [ERROR]Accept error: accept unixpacket /tmp/.s.GPMC.sock: accept4: too many open files

The error message contributes to ~99.9% of the file: 

wc -l agent.2019-11-19_112826.log
1406762139 agent.2019-11-19_112826.log
grep -c "too many open files" agent.2019-11-19_112826.log
1406713968

This issue may be attributed to a low limit on open files.

All servers had 262k set to the following limit:

open files                      (-n) 262144

We recommend 524288 as per the GPDB documentation.

Here are the recommended limits on open files:

• soft nofile 524288
• hard nofile 524288
• soft nproc 131072
• hard nproc 131072