Pivotal HDB PXF service fails to read AVRO binary when snappy compression is enabled
search cancel

Pivotal HDB PXF service fails to read AVRO binary when snappy compression is enabled

book

Article ID: 294910

calendar_today

Updated On:

Products

Services Suite

Issue/Introduction

Symptoms:

When scanning avro binary file with snappy compression HDB returns the following error

ERROR:  remote component error (500) from '172.28.17.50:51200':  type  Exception report   message   Servlet execution threw an exception    description   The server encountered an internal error that prevented it from fulfilling this request.    exception   javax.servlet.ServletException: Servlet execution threw an exception (libchurl.c:848)  (seg2 slice1 hdw2.phd.local:40000 pid=25857) (cdbdisp.c:1571)
DETAIL:  External table snappy_users

On one of the datanodes you will see the following stack trace in the /var/log/gphd/pxf/localhost.*.log

26-Nov-2014 14:59:27.588 SEVERE [tomcat-http--6] org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service() for servlet [PXF REST Service] in context with path [/pxf] threw exception [Servlet execution threw an exception] with root cause
 java.lang.ClassNotFoundException: org.xerial.snappy.Snappy
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1720)
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571)
        at org.apache.avro.file.SnappyCodec.decompress(SnappyCodec.java:58)
        at org.apache.avro.file.DataFileStream$DataBlock.decompressUsing(DataFileStream.java:343)
        at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:199)
        at org.apache.avro.mapred.AvroRecordReader.next(AvroRecordReader.java:67)
        at org.apache.avro.mapred.AvroRecordReader.next(AvroRecordReader.java:34)
        at com.pivotal.pxf.plugins.hdfs.AvroFileAccessor.readNextObject(AvroFileAccessor.java:60)
        at com.pivotal.pxf.service.ReadBridge.getNext(ReadBridge.java:61)
        at com.pivotal.pxf.service.rest.BridgeResource$1.write(BridgeResource.java:107)
        at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:71)
        at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:57)
        at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:306)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1437)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:744)

Environment


Cause

The pxf service does not have snappy 1.0.4.1 in its classpath

Resolution

  1. Ifhbase is installed we can add snappy 1.0.4.1 jar file formhbase into thepxf-service classpath. open file /etc/gphd/pxf/conf/pxf-public.classpath and add the following line 
    /usr/lib/gphd/hbase/lib/snappy-java-*.jar
    
 
  2. Restart Pivotal Hadoop sopxf-service will pick up the changes 
    icm_client restart -l phd_cluster
    
 
  3. Connect to HDB and verify external table scan works for avro data with snappy compression


Powered by Wolken Software