PXF server error : Could not find any valid local directory for s3ablock-0001-

Products

• VMware Tanzu Greenplum
• VMware Tanzu Greenplum / Gemfire
• VMware Tanzu Data Suite
• VMware Tanzu Data Suite

Issue/Introduction

When inserting data into a writable external table on S3 filesystem using Platform eXtension Framework (PXF) the following error can be reported:

ERROR:  PXF server error : Could not find any valid local directory for s3ablock-0001-  (seg0 10.10.10.11:6000 pid=385784)

or

ERROR:  PXF server error : No space available in any of the local directories.  (seg1 10.10.10.11:6523 pid=1360364)

Cause

S3 client may need to buffer data locally before writing to the S3 server. By default it will use When writing data to an S3 filesystem

Resolution

Specify a directory which PXF can use to store temporary data. By default it will use /tmp. To specify a different location with more space use the following parameter in the PXF configuration file for the server. It is possible to specify multiple locations.

<property>
  <name>fs.s3a.buffer.dir</name>
  <value>/data1/s3_buffer/s3a,/data2/s3_buffer/s3a</value>
  <description>Comma separated list of directories that will be used to buffer file
    uploads to. No effect if fs.s3a.fast.upload is true.</description>
</property>

Additional Information

See Apache Hadoop documentation for more information on the default values.