Error "gpfdist: "HTTP/1.0 408 time out ..." reported from the segment

Products

• VMware Tanzu Greenplum

Issue/Introduction

Symptoms:

When loading data into a HDB/GPDB cluster with gpfdist will fail with the following error message reported from segment instance.

ERROR: http response code 408 from gpfdist (gpfdist://etlhost:8090/hawq_mig_pipe): HTTP/1.0 408 time out (url.c:290) (seg26 hdw3.gphd.local:40006 pid=167513) (cdbdisp.c:1571)

A similar timeout message can be seen in gpfdist.log:

2015-07-02 15:48:37 85869 WARN [0:2:0:8] HTTP ERROR: ::ffff:10.81.103.91 - 408 time out

Environment

Cause

When executing a request, such as a gpfdist-based writable external table, the gpfdist server reads data from segment instances of the HDB/GPDB cluster. When there is a lot of network traffic on the system or a gpfdist/segment instance is busy, the gpfdist/segment connection fails to be established before a timeout occurs.

A timer is set on the gpfdist side to control the time allowed for a segment instance to establish a connection to a gpfdist process. The default timeout value is 5 seconds. The timeout value allowed ranges from 2 to 600 seconds for GPDB (2~30 seconds for HDB).

Resolution

If the timeout is caused by heavy network traffic or a busy gpfdist/segment instance, it can solved by increasing the timeout value with the "-t" option when starting gpfdist.

If the maximum allowed value is still not enough, then "-t 0" can be tried to disable the timeout.

Note: "-t 0" is not mentioned in manual, but it does exist, as shown below.
 

Greenplum Database

[gpadmin@mdw ~]$ gpfdist -t 10000
Error: -t timeout must be between 2 and 600, or 0 for no timeout

HAWQ Database

[gpadmin@hdm1 ~]$ gpfdist -t 1000
Error: -t timeout must be between 2 and 30, or 0 for no timeout