when access pxf external tables, receive error "HTTP status code is 500 but HTTP response string is empty"

Products

• VMware Tanzu Greenplum

Issue/Introduction

Sometimes, when access pxf external tables, it may report error :

ERROR:  PXF server error : HTTP status code is 500 but HTTP response string is empty  (seg0 slice1 000.000.000.000:6871 pid=478376)
CONTEXT:  External table tabb1, line 1 of file pxf://tabb1.csv?PROFILE=HdfsTextSimple&SERVER=default
gpadmin=#

Environment

greenplum 6.x, 7.x

pxf 6.x

Cause

From the pxf log of the related segment host, it may report more detailed error message:

 ERROR [:   ] 151193 --- [-exec-4180] o.a.c.c.C.[.[.[.[dispatcherServlet]      : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.core.task.TaskRejectedException: Executor [org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor$1@9e67eb1[Running, pool size = 200, active threads = 200, queued tasks = 0, completed tasks = 664286]] did not accept task: org.springframework.web.context.request.async.WebAsyncManager$$hostname with root cause

java.util.concurrent.RejectedExecutionException: Task org.greenplum.pxf.service.spring.PxfContextMdcLogEnhancerDecorator$$hostname rejected from org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor$1@9e67eb1[Running, pool size = 200, active threads = 200, queued tasks = 0, completed tasks = 664286]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) ~[?:1.8.0_312]
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_312]
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) ~[?:1.8.0_312]
        at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor$1.execute(ThreadPoolTaskExecutor.java:257) ~[spring-context-5.3.18.jar!/:5.3.18]
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) ~[?:1.8.0_312]
        at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.submit(ThreadPoolTaskExecutor.java:353) ~[spring-context-5.3.18.jar!/:5.3.18]
        at org.greenplum.pxf.service.spring.PxfThreadPoolTaskExecutor.submit(PxfThreadPoolTaskExecutor.java:34) ~[classes!/:6.6.0]
        at org.springframework.web.context.request.async.WebAsyncManager.startCallableProcessing(WebAsyncManager.java:333) ~[spring-web-5.3.18.jar!/:5.3.18]
        at org.springframework.web.context.request.async.WebAsyncManager.startCallableProcessing(WebAsyncManager.java:267) ~[spring-web-5.3.18.jar!/:5.3.18]
        at org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBodyReturnValueHandler.handleReturnValue(StreamingResponseBodyReturnValueHandler.java:94) ~[spring-webmvc-5.3.18.jar!/:5.3.18]

This error means this host's pxf service has reached it's tomcat thread limit of 200.  By default pxf has tomcat thread limit of 200. max pool size limit 200. This could happen when there are too much pxf query running, or one query used too much pxf external tables.

Resolution

To avoid this error, there are two directions:
1. change pxf setting from $PXF_BASE/conf/pxf-application.properties. But this could more memory usage from pxf service. 

# pxf.max.threads=200
# pxf.task.pool.max-size=200

change to :
 pxf.max.threads=400
 pxf.task.pool.max-size=400

2. Avoid use too many pxf external table in one query or running many pxf query at the same time.