hadoop流使用shell脚本：reducer失败，错误为：没有这样的文件或目录

hwazgwia  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(424)

我使用的是一个10节点的hdp集群，我试图在bash上使用shell脚本运行一个简单的wordcount作业。

yarn jar /usr/hdp/2.6.5.0-292/hadoop-mapreduce/hadoop-streaming-2.7.3.2.6.5.0-292.jar \
    -mapper 'wc -l' \
    -reducer './reducer_wordcount.sh' \
    -file /home/pathirippilly/map_reduce_jobs/shell_scripts/reducer_wordcount.sh \
    -numReduceTasks 1 \
    -input /user/pathirippilly/cards/smalldeck.txt \
    -output /user/pathirippilly/mapreduce_jobs/output_shell

这里reducer\u wordcount.sh是reducer shell脚本，可以在我的本地目录/home/pathirippilly/map\u reduce\u jobs/shell\u scripts中找到
smalldeck.txt是hadoop目录/user/pathirippilly/cards上的输入文件
/user/pathirippilly/mapreduce\u jobs/output\u shell是输出目录
我使用的hadoop版本是hadoop2.7.3.2.6.5.0-292
我正在Yarn模式下运行上面的map reduce作业
您的wordcount.sh有：


# ! /user/bin/env bash

    awk '{line_count += $1} END  { print line_count }'

当我在我的集群上运行这个程序时，我得到了以下关于reducer\u wordcount.sh的错误

Error: java.lang.RuntimeException: Error in configuring object
            at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
            at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
            at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
            at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:410)
            at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
            at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
            at java.security.AccessController.doPrivileged(Native Method)
            at javax.security.auth.Subject.doAs(Subject.java:422)
            at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
            at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
    Caused by: java.lang.reflect.InvocationTargetException
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:498)
            at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
            ... 9 more
    Caused by: java.lang.RuntimeException: configuration exception
            at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:222)
            at org.apache.hadoop.streaming.PipeReducer.configure(PipeReducer.java:67)
            ... 14 more
    Caused by: java.io.IOException: Cannot run program "/hdp01/hadoop/yarn/local/usercache/pathirippilly/appcache/application_1533622723243_17238/container_e38_1533622723243_17238_01_000004/./reducer_wordcount.sh": error=2, No such file or directory
            at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
            at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:209)
            ... 15 more
    Caused by: java.io.IOException: error=2, No such file or directory
            at java.lang.UNIXProcess.forkAndExec(Native Method)
            at java.lang.UNIXProcess.<init>(UNIXProcess.java:248)
            at java.lang.ProcessImpl.start(ProcessImpl.java:134)
            at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)

如果我直接运行与下面命令行commad相同的reducer脚本，它就可以工作了

yarn jar /usr/hdp/2.6.5.0-292/hadoop-mapreduce/hadoop-streaming.jar \
    -mapper 'wc -l' \
    -reducer "awk '{line_count += \$1} END  { print line_count }'" \
    -numReduceTasks 1 \
    -input /user/pathirippilly/cards/smalldeck.txt \
    -output /user/pathirippilly/mapreduce_jobs/output_shell

我对hadoop流媒体还很陌生，希望能得到帮助。完整的错误堆栈如下所示：

18/09/09 10:10:02 WARN streaming.StreamJob: -file option is deprecated, please use generic option -files instead.
    packageJobJar: [reducer_wordcount.sh] [/usr/hdp/2.6.5.0-292/hadoop-mapreduce/hadoop-streaming-2.7.3.2.6.5.0-292.jar] /var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir/streamjob8506373101127930734.jar tmpDir=null
    18/09/09 10:10:03 INFO client.RMProxy: Connecting to ResourceManager at rm01.itversity.com/172.16.1.106:8050
    18/09/09 10:10:03 INFO client.AHSProxy: Connecting to Application History server at rm01.itversity.com/172.16.1.106:10200
    18/09/09 10:10:03 INFO client.RMProxy: Connecting to ResourceManager at rm01.itversity.com/172.16.1.106:8050
    18/09/09 10:10:03 INFO client.AHSProxy: Connecting to Application History server at rm01.itversity.com/172.16.1.106:10200
    18/09/09 10:10:05 INFO mapred.FileInputFormat: Total input paths to process : 1
    18/09/09 10:10:06 INFO mapreduce.JobSubmitter: number of splits:2
    18/09/09 10:10:07 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1533622723243_17238
    18/09/09 10:10:08 INFO impl.YarnClientImpl: Submitted application application_1533622723243_17238
    18/09/09 10:10:08 INFO mapreduce.Job: The url to track the job: http://rm01.itversity.com:19288/proxy/application_1533622723243_17238/
    18/09/09 10:10:08 INFO mapreduce.Job: Running job: job_1533622723243_17238
    18/09/09 10:10:14 INFO mapreduce.Job: Job job_1533622723243_17238 running in uber mode : false
    18/09/09 10:10:14 INFO mapreduce.Job:  map 0% reduce 0%
    18/09/09 10:10:19 INFO mapreduce.Job:  map 100% reduce 0%
    18/09/09 10:10:23 INFO mapreduce.Job: Task Id : attempt_1533622723243_17238_r_000000_0, Status : FAILED
    Error: java.lang.RuntimeException: Error in configuring object
            at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
            at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
            at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
            at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:410)
            at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
            at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
            at java.security.AccessController.doPrivileged(Native Method)
            at javax.security.auth.Subject.doAs(Subject.java:422)
            at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
            at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
    Caused by: java.lang.reflect.InvocationTargetException
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:498)
            at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
            ... 9 more
    Caused by: java.lang.RuntimeException: configuration exception
            at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:222)
            at org.apache.hadoop.streaming.PipeReducer.configure(PipeReducer.java:67)
            ... 14 more
    Caused by: java.io.IOException: Cannot run program "/hdp01/hadoop/yarn/local/usercache/pathirippilly/appcache/application_1533622723243_17238/container_e38_1533622723243_17238_01_000004/./reducer_wordcount.sh": error=2, No such file or directory
            at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
            at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:209)
            ... 15 more
    Caused by: java.io.IOException: error=2, No such file or directory
            at java.lang.UNIXProcess.forkAndExec(Native Method)
            at java.lang.UNIXProcess.<init>(UNIXProcess.java:248)
            at java.lang.ProcessImpl.start(ProcessImpl.java:134)
            at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
            ... 16 more
hadoopmapreducehadoop2hadoop-streaming

1条答案

按热度按时间
kqhtkvqz

kqhtkvqz1#

请参阅使文件可用于任务和打包文件以提交作业
基本上，您只需要脚本的文件名，而不需要路径

-reducer 'reducer_wordcount.sh' -file /local/path/to/reducer_wordcount.sh

确保文件是可执行的

chmod +x /local/path/to/reducer_wordcount.sh

可以选择使用 # 标记，但您的本地脚本名称与reducer文件相同，因此这不是必需的。
你还需要把这个修好 #!/usr/bin/env bash （顺便说一句，你的Map器和减缩器在做同样的事情，数行，不一定是“单词”）

赞(0）分享  回复(0）举报  2021-05-29
我来回答

相关问题

    查看更多
    微信公众号

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com