ETL data loading failed with error "gpfdist error: server closed connection" in Tanzu Greenplum

Products

• VMware Tanzu Greenplum
• VMware Tanzu Data Suite
• VMware Tanzu Data Suite

Issue/Introduction

Error will always show when loading large amount of data.

Error message:

gpfdist error: server closed connection 

When loading smaller amount of data, the same job works fine. From the ETL host's /var/log/messages (which runs the gpfdist process), there are  many errors:

gpfdist) of user 1234 killed by SIGABRT - ignoring (repeated crash)
gpfdist) of user 1234 killed by SIGABRT - ignoring (repeated crash)
(gpfdist) of user 1234 killed by SIGABRT - dumping core
(gpfdist) of user 1234 killed by SIGABRT - ignoring (repeated crash)
(gpfdist) of user 1234 killed by SIGABRT - ignoring (repeated crash)
(gpfdist) of user 1234 killed by SIGABRT - ignoring (repeated crash)
(gpfdist) of user 1234 killed by SIGABRT - dumping core
(gpfdist) of user 1234 killed by SIGABRT - ignoring (repeated crash)
(gpfdist) of user 1234 killed by SIGABRT - ignoring (repeated crash)
(gpfdist) of user 1234 killed by SIGABRT - ignoring (repeated crash)
(gpfdist) of user 1234 killed by SIGABRT - dumping core 

 

Environment

Product Version: 5.27

Resolution

From the user environment setting, there is a OS setting:  gpfdist_watchdog_timer. The previous setting is 300, after set a higher value, the ETL data loading works fine.

Related document Troubleshooting gpfdist
 

If the gpfdist utility hangs with no read or write activity occurring, you can generate a core dump the next time a hang occurs to help debug the issue. Set the environment variable GPFDIST_WATCHDOG_TIMER to the number of seconds of no activity to wait before gpfdist is forced to exit. When the environment variable is set and gpfdist hangs, the utility aborts after the specified number of seconds, creates a core dump, and sends abort information to the log file.

This example sets the environment variable on a Linux system so that gpfdist exits after 300 seconds (5 minutes) of no activity.
export GPFDIST_WATCHDOG_TIMER=300