Cannot use PXF to query Hive tables on a Kerberized environment

Products

• VMware Tanzu Greenplum

Issue/Introduction

After upgrading GPDB from 5.13 to 5.16, external tables using PXF with Hive stopped working.

Hive is in a Kerberized HDP cluster, however we are able to query HDFS tables in the same HDP cluster.
 

When querying Hive data, we are getting the following error:

ckdata=# select * from gp_external_table_test limit 1;
ERROR: remote component error (500) from '127.0.0.1:5888': type Exception report message Failed connecting to Hive MetaStore service: Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused) description The server encountered an internal error that prevented it from fulfilling this request. exception java.lang.RuntimeException: Failed connecting to Hive MetaStore service: Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused) (libchurl.c:944) (seg0 slice1 169.254.4.4:6000 pid=25626) (cdbdisp.c:254)
DETAIL: External table gp_external_table_test

Master log shows:

2019-03-19 09:52:11.851237 MSK,"gpadmin","user",p25450,th945883008,"[local]",,2019-03-19 09:51:57 MSK,0,con207598,cmd3,seg-1,,dx275278,,sx1,"ERROR","XX000","remote component error (500) from '127.0.0.1:5888':  type  Exception report   message   Index: 18, Size: 18    description   The server encountered an internal error that prevented it from fulfilling this 
request.    exception   java.io.IOException: Index: 18, Size: 18 (libchurl.c:944)  (seg0 slice1 169.254.4.4:6000 pid=17960) (cdbdisp.c:254)","External table rtl_txn_header",,,,,"select * from rtl_txn_header limit 1;",0,,"cdbdisp.c",254,"Stack trace:
1    0x95ee9b postgres errstart (elog.c:521)
2    0x9e80c0 postgres cdbdisp_finishCommand (cdbdisp.c:252)
3    0x9e831a postgres CdbDispatchHandleError (discriminator 2)
4    0x6af9c7 postgres mppExecutorCleanup (execUtils.c:2092)
5    0x699a70 postgres ExecutorRun (execMain.c:958)
6    0x83c979 postgres <symbol not found> (pquery.c:1161)
7    0x83e5f5 postgres PortalRun (pquery.c:984)
8    0x839146 postgres <symbol not found> (postgres.c:1783)
9    0x83b351 postgres PostgresMain (postgres.c:4962)
10   0x7da8e5 postgres <symbol not found> (postmaster.c:6406)
11   0x7dd0ba postgres PostmasterMain (postmaster.c:1533)
12   0x4cb077 postgres main (main.c:206)
13   0x7ff833b383d5 libc.so.6 __libc_start_main + 0xf5
14   0x4cb5cc postgres <symbol not found> + 0x4cb5cc

On the pxf-service.log we can see:

2019-03-19 09:56:36.0315 ERROR tomcat-http--24 org.greenplum.pxf.service.rest.BridgeResource - Exception thrown when streaming
java.lang.IndexOutOfBoundsException: Index: 18, Size: 18
        at java.util.ArrayList.rangeCheck(ArrayList.java:657)
        at java.util.ArrayList.get(ArrayList.java:433)
        at org.greenplum.pxf.plugins.hive.HiveResolver.traverseStruct(HiveResolver.java:473)
        at org.greenplum.pxf.plugins.hive.HiveResolver.getFields(HiveResolver.java:112)
        at org.greenplum.pxf.service.bridge.ReadBridge.makeOutput(ReadBridge.java:76)
        at org.greenplum.pxf.service.bridge.ReadBridge.getNext(ReadBridge.java:106)
        at org.greenplum.pxf.service.rest.BridgeResource$1.write(BridgeResource.java:139)
        at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:71)
        at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:57)
        at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:306)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1437)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.greenplum.pxf.service.servlet.SecurityServletFilter.lambda$doFilter$0(SecurityServletFilter.java:105)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
        at org.greenplum.pxf.service.servlet.SecurityServletFilter.doFilter(SecurityServletFilter.java:120)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:957)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:423)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1079)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:620)
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)

This issue is reproducible internally.

Environment

Product Version: 5.16

Resolution

A bug was identified for version 5.16 and a fix was provided in version 5.19.