Hive query fails with error message "ArrayIndexOutOfBoundsException"
Article ID: 294821
Updated On:
Products
Services Suite
Issue/Introduction
Symptoms:
Hive queries with TEZ fail with the following error message:
Vertex failed, vertexName=Map 1, vertexId=vertex_1465366512401_87702_1_01, diagnostics=[Task failed, taskId=task_1465366512401_87702_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"itemnum":"11038949","location":"4000","conditioncode":"B","condrate":75.0,"itemsetid":"WMS_IT05","siteid":"FVN-NU62","stdcost":0,"avgcost":0,"lastcost":0,"glaccount":null,"controlacc":"GBGN.0000000000.CTR.0000000000","shrinkageacc":"GBGN.0000000000.CTR.0000000000","invcostadjacc":"GBGN.0000000000.CTR.0000000000","orgid":"UKS","invcostid":1.06719823E8,"rowstamp":"623292900","change_date":623292900,"load_ts":"2016-02-03 11:04:01","row__id":{"transactionid":640,"bucketid":2,"rowid":13770426}}
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:186)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:138)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"itemnum":"11038949","location":"4000","conditioncode":"B","condrate":75.0,"itemsetid":"WMS_IT05","siteid":"FVN-NU62","stdcost":0,"avgcost":0,"lastcost":0,"glaccount":null,"controlacc":"GBGN.0000000000.CTR.0000000000","shrinkageacc":"GBGN.0000000000.CTR.0000000000","invcostadjacc":"GBGN.0000000000.CTR.0000000000","orgid":"UKS","invcostid":1.06719823E8,"rowstamp":"623292900","change_date":623292900,"load_ts":"2016-02-03 11:04:01","row__id":{"transactionid":640,"bucketid":2,"rowid":13770426}}
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:294)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:163)
... 13 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"itemnum":"11038949","location":"4000","conditioncode":"B","condrate":75.0,"itemsetid":"WMS_IT05","siteid":"FVN-NU62","stdcost":0,"avgcost":0,"lastcost":0,"glaccount":null,"controlacc":"GBGN.0000000000.CTR.0000000000","shrinkageacc":"GBGN.0000000000.CTR.0000000000","invcostadjacc":"GBGN.0000000000.CTR.0000000000","orgid":"UKS","invcostid":1.06719823E8,"rowstamp":"623292900","change_date":623292900,"load_ts":"2016-02-03 11:04:01","row__id":{"transactionid":640,"bucketid":2,"rowid":13770426}}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:503)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83)
... 16 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ArrayIndexOutOfBoundsException
at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:397)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:493)
... 17 more
Caused by: java.lang.ArrayIndexOutOfBoundsException
at org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter$Buffer.write(DefaultSorter.java:582)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
at org.apache.tez.runtime.library.common.serializer.TezBytesWritableSerialization$TezBytesWritableSerializer.serialize(TezBytesWritableSerialization.java:123)
at org.apache.tez.runtime.library.common.serializer.TezBytesWritableSerialization$TezBytesWritableSerializer.serialize(TezBytesWritableSerialization.java:110)
at org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.collect(DefaultSorter.java:275)
at org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.write(DefaultSorter.java:185)
at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput$1.write(OrderedPartitionedKVOutput.java:127)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor$TezKVOutputCollector.collect(TezProcessor.java:211)
at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.collect(ReduceSinkOperator.java:534)
at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:380)
... 21 more
Environment
Cause
This error is caused by a known bug reported in Apache Tez: TEZ-2732 . This happens when "tez.runtime.io.sort.mb" is set to 2047 (max value allowed).
Resolution
The workaround is to set tez.runtime.io.sort.mb to something lower. The recommendation is 1800.
This change should be done through Ambari in Tez service.
Additional Information
For further information, please access the following resources:
Feedback
Yes
No
Powered by
