hive mapreduce作业提交失败“目标是目录”

8qgya5xd  于 2021-06-04  发布在  Hadoop
关注(0)|答案(1)|浏览(301)

我一直在玩hadoop和它的姊妹项目,一路上我遇到了一些问题,但我终于找到了一个我找不到答案的问题:
我有一个配置单元表作为制表符分隔的文本文件存储在hdfs上。我可以在表上做一个基本的select,但是只要我把查询弄得稍微复杂一点,hive就会把它变成一个map reduce作业,它会失败,并出现下面的堆栈跟踪
13/11/29 08:31:00错误security.usergroupinformation:priviledgedactionexception as:hduser(auth:simple) cause:java.io.ioexception:target/tmp/hadoop->>yarn/staging/hduser/.staging/job\u 1385633903169\u 0013/libjars/lib/lib是目录13/11/29 08:31:00 error security.usergroupinformation:priviledgedactionexception as:hduser(auth:simple) cause:java.io.ioexception:target/tmp/hadoop-yarn/staging/hduser/.staging/job\u 1385633903169\u 0013/libjars/lib/lib是目录java.io.ioexception:target/tmp/hadoop-yarn/staging/hduser/.staging/job\u 1385633903169\u 0013/libjars/lib是org.apache.hadoop.fs.fileutil.checkdest(fileutil)中的目录。java:500)在org.apache.hadoop.fs.fileutil.checkdest(fileutil。java:502)在org.apache.hadoop.fs.fileutil.copy(fileutil。java:348)在org.apache.hadoop.fs.fileutil.copy(fileutil。java:338)在org.apache.hadoop.fs.fileutil.copy(fileutil。java:289)在org.apache.hadoop.mapreduce.jobsubmitter.copyremotefiles(jobsubmitter。java:139)在org.apache.hadoop.mapreduce.jobsubmitter.copyandconfigurefiles(jobsubmitter。java:212)在org.apache.hadoop.mapreduce.jobsubmitter.copyandconfigurefiles(jobsubmitter。java:300)在org.apache.hadoop.mapreduce.jobsubmitter.submitjobinternal(jobsubmitter。java:387)在org.apache.hadoop.mapreduce.job$10.run(job。java:1268)在org.apache.hadoop.mapreduce.job$10.run(作业。java:1265)位于javax.security.auth.subject.doas(subject)的java.security.accesscontroller.doprivileged(本机方法)。java:415)在org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation。java:1491)在org.apache.hadoop.mapreduce.job.submit(作业。java:1265)在org.apache.hadoop.mapred.jobclient$1.run(jobclient。java:562)在org.apache.hadoop.mapred.jobclient$1.run(jobclient。java:557)位于javax.security.auth.subject.doas(subject)的java.security.accesscontroller.doprivileged(本机方法)。java:415)在org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation。java:1491)在org.apache.hadoop.mapred.jobclient.submitjobinternal(jobclient。java:557)在org.apache.hadoop.mapred.jobclient.submitjob(jobclient。java:548)在org.apache.hadoop.hive.ql.exec.mr.execdriver.execute(execdriver。java:425)位于org.apache.hadoop.hive.ql.exec.mr.mapredtask.execute(mapredtask)。java:144)在org.apache.hadoop.hive.ql.exec.task.executetask(任务。java:151)位于org.apache.hadoop.hive.ql.exec.taskrunner.runsequential(taskrunner)。java:65)在org.apache.hadoop.hive.ql.driver.launchtask(driver。java:1414)在org.apache.hadoop.hive.ql.driver.execute(driver。java:1192)在org.apache.hadoop.hive.ql.driver.runinternal(driver。java:1020)在org.apache.hadoop.hive.ql.driver.run(驱动程序。java:888)在org.apache.hadoop.hive.cli.clidriver.processlocalcmd(clidriver。java:259)在org.apache.hadoop.hive.cli.clidriver.processcmd(clidriver。java:216)在org.apache.hadoop.hive.cli.clidriver.processline(clidriver。java:413)在org.apache.hadoop.hive.cli.clidriver.executedriver(clidriver。java:781)在org.apache.hadoop.hive.cli.clidriver.run(clidriver。java:675)位于org.apache.hadoop.hive.cli.clidriver.main(clidriver。java:614)在sun.reflect.nativemethodaccessorimpl.invoke0(本机方法)在sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl)。java:57)在sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl。java:43)在java.lang.reflect.method.invoke(方法。java:606)在org.apache.hadoop.util.runjar.main(runjar。java:212)作业提交失败,出现异常“java.io.ioexception(target/tmp/hadoop/staging/hduser/.staging/job\u 1385633903169\u 0013/libjars/lib/lib是目录)'13/11/29 08:31:00 error exec.task:作业提交失败,出现异常'java.io.ioexception(target/tmp/hadoop-yarn/staging/hduser/.staging/job__0013/libjars/lib/lib是目录)'java.io.ioexception:target/tmp/hadoop-yarn/staging/hduser/.staging/job__0013/libjars/lib是org.apache.hadoop.fs.fileutil.checkdest(fileutil。java:500)在org.apache.hadoop.fs.fileutil.checkdest(fileutil。java:502)在org.apache.hadoop.fs.fileutil.copy(fileutil。java:348)在org.apache.hadoop.fs.fileutil.copy(fileutil。java:338)在org.apache.hadoop.fs.fileutil.copy(fileutil。java:289)在org.apache.hadoop.mapreduce.jobsubmitter.copyremotefiles(jobsubmitter。java:139)在org.apache.hadoop.mapreduce.jobsubmitter.copyandconfigurefiles(jobsubmitter。java:212)在org.apache.hadoop.mapreduce.jobsubmitter.copyandconfigurefiles(jobsubmitter。java:300)在org.apache.hadoop.mapreduce.jobsubmitter.submitjobinternal(jobsubmitter。java:387)在org.apache.hadoop.mapreduce.job$10.run(作业。java:1268)在org.apache.hadoop.mapreduce.job$10.run(job。java:1265)位于javax.security.auth.subject.doas(subject)的java.security.accesscontroller.doprivileged(本机方法)。java:415)在org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation。java:1491)在org.apache.hadoop.mapreduce.job.submit(作业。java:1265)在org.apache.hadoop.mapred.jobclient$1.run(jobclient。java:562)在org.apache.hadoop.mapred.jobclient$1.run(jobclient。java:557)位于javax.security.auth.subject.doas(subject)的java.security.accesscontroller.doprivileged(本机方法)。java:415)在org.apache.hadoop.security.usergroupinformation.doas(用户组信息。java:1491)在org.apache.hadoop.mapred.jobclient.submitjobinternal(jobclient。java:557)在org.apache.hadoop.mapred.jobclient.submitjob(jobclient。java:548)在org.apache.hadoop.hive.ql.exec.mr.execdriver.execute(execdriver。java:425)在org.apache.hadoop.hive.ql.exec.mr.mapredtask.execute(mapredtask)。java:144)在org.apache.hadoop.hive.ql.exec.task.executetask(任务。java:151)位于org.apache.hadoop.hive.ql.exec.taskrunner.runsequential(taskrunner)。java:65)在org.apache.hadoop.hive.ql.driver.launchtask(driver。java:1414)在org.apache.hadoop.hive.ql.driver.execute(driver。java:1192)在org.apache.hadoop.hive.ql.driver.runinternal(驱动程序。java:1020)在org.apache.hadoop.hive.ql.driver.run(driver。java:888)在org.apache.hadoop.hive.cli.clidriver.processlocalcmd(clidriver。java:259)在org.apache.hadoop.hive.cli.clidriver.processcmd(clidriver。java:216)在org.apache.hadoop.hive.cli.clidriver.processline(clidriver。java:413)在org.apache.hadoop.hive.cli.clidriver.executedriver(clidriver。java:781)在org.apache.hadoop.hive.cli.clidriver.run(clidriver。java:675)位于org.apache.hadoop.hive.cli.clidriver.main(clidriver。java:614)在sun.reflect.nativemethodaccessorimpl.invoke0(本机方法)在sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl)。java:57)在sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl。java:43)在java.lang.reflect.method.invoke(方法。java:606)在org.apache.hadoop.util.runjar.main(runjar。java:212)
有问题的文件夹确实存在于dfs上,至少在“/tmp/hadoop/staging”部分,并且无论我将其权限设置为什么,hive或hadoop都会在作业提交时重置它们。真正令人担忧的是,完整路径似乎是一个生成的文件夹名,那么为什么软件对它自己生成的东西有问题呢?为什么路径是目录会有问题?它应该是什么呢?
编辑:这是我正在处理的表和我尝试运行的查询:query: select * from hive_flow_details where node_id = 100 limit 10; 表格:
列名称数据类型注解id bigint none
流量\版本\ id int无
节点id int无
节点名称字符串无
请记住,这发生在我尝试的任何带有where子句的查询上,因为hive将其转换为mr job。

aemubtdh

aemubtdh1#

我最终在这个问题上说对了。我在我清理的类路径中发现了冲突的jar,从那时起我就没有任何问题了。

相关问题