gpcopy failed due to error "data line too long. likely due to invalid csv data"

Products

• VMware Tanzu Greenplum

Issue/Introduction

In this case, gpcopy failed with the error below when migrating a table from the source cluster to the destination cluster. It looks related to seg10 in this case:

20201124:10:20:38 gpcopy:gpadmin:<hostname>:029987-[ERROR]:-[Worker 0] sendData() failed to copy table "table_name" => "<table_name>" 
20201124:10:20:38 gpcopy:gpadmin:<hostname>:029987-[CRITICAL]:-pq: Error from segment 10

From the seg10 log of the destination cluster, we can see a key error message:

ERROR: |22P04|data line too long. likely due to invalid csv data|||||

According to the error message above, the issue occurs when a CSV line is over 1MB as the GUC gp_max_csv_line_length defaults to 1MB.

Environment

Product Version: 5.15

Resolution

To fix this issue, we can increase the value of the GUC gp_max_csv_line_length guc to allow for longer CSV lines like 5 MB and then reload the configuration to make it take effect.