gpfdist fails when the server closes the connection due to premature End of File errors

Products

• VMware Tanzu Greenplum

Issue/Introduction

When running gpfdist, the following error is seen in the Greenplum log when gpfdist fails:

ERROR: gpfdist server closed connection (seg30 slice1 x.x.x.x:xxxxx pid=5896)(Hint The root cause is likely to be an overload of the ETL host or a temporary network glitch between the database and the ETL host causing the connection between the gpfdist and database to disconnect.;

This can be due to communication delays between the server running gpfdist and the Greenplum cluster. When checking the gpfdist logs, there are many EOF failures like the ones below:

2021-04-29 13:31:00 20132 INFO [0:821823:54:97] session already ended. return empty response (OK) 
2021-04-29 13:31:00 20132 INFO [0:821823:54:97] HTTP EMPTY: ::ffff:x.x.x.x GET 
2021-04-29 13:31:00 20132 INFO [0:821823:54:97] sent EOF: succeed 
2021-04-29 13:31:00 20132 INFO [0:821823:54:97] request end 
2021-04-29 13:31:00 20132 INFO [0:821823:54:97] detach segment request from session 2021-04-29 13:31:00 20132 INFO [0:821823:54:97] SSL cleanup and free 
2021-04-29 13:31:00 20132 INFO [0:821823:54:97] successfully shutdown socket 
2021-04-29 13:31:00 20132 WARN [0:821823:54:97] gpfdist read unexpected data after shutdown 
2021-04-29 13:31:00 20132 INFO [0:821823:54:97] unsent bytes: 0 (-1 means not supported) 
2021-04-29 13:31:00 20132 INFO [0:821823:54:97] successfully closed socket

Environment

Product Version: 6.11

Resolution

This is due to network slowness between whichever server is running gpfdist and the Greenplum cluster causing timeouts that lead to premature EOFs eventually killing the gpfdist command.

Adding the '-w' switch into the gpfdist command with an acceptable value, such as 'gpfdist -w 60' will add an extra 60 second wait time to the gpfdist command before it closes each session.

This value is discussed more in the Utility Guide linked below:
 

•  https://docs.vmware.com/en/VMware-Greenplum/7/greenplum-database/utility_guide-ref-gpfdist.html