在配置单元上重建索引失败

yzxexxkh 于 2021-05-29 发布在 Hadoop

关注(0)|答案(2)|浏览(452)

我正在使用hortonworks沙盒探索hive的一些功能，并使用sqoop将一个表导入其中。
因为它是一个相当大的表，所以我尝试在其中一列上创建索引。它让我加上“ WITH DEFERRED REBUILD “我做的选项，索引被创建。
但当我试图重建索引时，作业失败，日志如下。
日志

Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: java.lang.NullPointerException
    at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
    at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
    at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:337)
    at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
    at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
    at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
    at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.RuntimeException: java.lang.NullPointerException
    at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:192)
    at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.<init>(TezGroupedSplitsInputFormat.java:131)
    at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:97)
    at org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:149)
    at org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:80)
    at org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:614)
    at org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:593)
    at org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:141)
    at org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:109)
    at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:370)
    at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:127)
    at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:147)
    ... 14 more

Caused by: java.lang.NullPointerException
    at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:270)
    at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:234)
    at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:189)
    ... 25 more

hadoop Hive bigdata

来源：https://stackoverflow.com/questions/33079345/rebuild-index-on-hive-fails

2条答案

按热度按时间

5gfr0r5j1#

这个问题实际上与tez中错误配置的内存参数有关。
这就是为什么改用mr解决了问题。
一篇关于tez内存的好文章在调试这个问题时非常有用。
https://community.hortonworks.com/content/kbentry/14309/demystify-tez-tuning-step-by-step.html

赞(0）回复(0）举报 2021-05-30

pgpifvop2#

我也遇到了同样的问题。我找到了解决办法

set hive.execution.engine=mr;
alter index index_name on table_name rebuild;
set hive.execution.engine=tez;

使用mr运行索引的重建对我来说没有问题。我不知道为什么泰兹会失败。

赞(0）回复(0）举报 2021-05-30

我来回答

在配置单元上重建索引失败

2条答案

相关问题

热门标签

最新问答