rstudio服务器中的hadoop流失败，错误代码为1

vjhs03f7  于 2021-05-29  发布在  Hadoop
关注(0)|答案(0)|浏览(286)

我使用单节点。
我在sudor中安装了rmr2和hdfs
我在rstudio服务器上写了一些代码。
但是，它发生了错误。
我不知道怎么了。
谢谢你的阅读。如果有人帮助我，我会感激你的。

> library("rmr2", lib.loc="/usr/lib64/R/library")   
Please review your hadoop settings. See help(hadoop.settings)

> library("rhdfs", lib.loc="/usr/lib64/R/library")   
Loading required package: rJava    
HADOOP_CMD=/home/knu/hadoop/hadoop-2.7.3/bin/hadoop    
Be sure to run hdfs.init()

> hdfs.init()    
17/04/12 21:39:15 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

> detach("package:rmr2", unload=TRUE)    
> library("rmr2", lib.loc="/usr/lib64/R/library")    
Please review your hadoop settings. See help(hadoop.settings)

> small.ints <- to.dfs(1:10)
17/04/12 21:41:07 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
17/04/12 21:41:07 INFO compress.CodecPool: Got brand-new compressor [.deflate]

> from.dfs(small.ints)
17/04/12 21:41:25 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
17/04/12 21:41:25 INFO compress.CodecPool: Got brand-new decompressor [.deflate]

$key
NULL

$val
[1]  1  2  3  4  5  6  7  8  9 10

> result <- mapreduce(input = small.ints, 
                    map = function(k,v) cbind(v,v^2)  )

执行日志：

packageJobJar: [/tmp/hadoop-unjar5646463829062726981/] [] /tmp/streamjob3604004992263150530.jar tmpDir=null

17/04/12 21:41:38 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032

17/04/12 21:41:38 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032

17/04/12 21:41:39 INFO mapred.FileInputFormat: Total input paths to process : 1

17/04/12 21:41:39 INFO mapreduce.JobSubmitter: number of splits:2

17/04/12 21:41:39 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces

17/04/12 21:41:39 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1491995822063_0001

17/04/12 21:41:40 INFO impl.YarnClientImpl: Submitted application application_1491995822063_0001

17/04/12 21:41:40 INFO mapreduce.Job: The url to track the job: http://0.0.0.0:8089/proxy/application_1491995822063_0001/

17/04/12 21:41:40 INFO mapreduce.Job: Running job: job_1491995822063_0001

17/04/12 21:41:54 INFO mapreduce.Job: Job job_1491995822063_0001 running in uber mode : false

17/04/12 21:41:54 INFO mapreduce.Job:  map 0% reduce 0%

17/04/12 21:42:01 INFO mapreduce.Job: Task Id : attempt_1491995822063_0001_m_000001_0, Status : FAILED

Container [pid=12055,containerID=container_1491995822063_0001_01_000003] is running beyond virtual memory limits. Current usage: 109.6 MB of 1 GB physical memory used; 2.1 GB of 2.1 GB virtual memory used. Killing container.

Dump of the process-tree for container_1491995822063_0001_01_000003 :

    |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE

    |- 12073 12055 12055 12055 (java) 193 8 2153918464 27756 /home/knu/hadoop/jdk1.8.0_121/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx400M -Djava.io.tmpdir=/home/knu/hadoop/hadoop-2.7.3/data/yarn/nm-local-dir/usercache/knu/appcache/application_1491995822063_0001/container_1491995822063_0001_01_000003/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000003 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 39313 attempt_1491995822063_0001_m_000001_0 3 

    |- 12055 12053 12055 12055 (bash) 0 0 115847168 303 /bin/bash -c /home/knu/hadoop/jdk1.8.0_121/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN  -Xmx400M -Djava.io.tmpdir=/home/knu/hadoop/hadoop-2.7.3/data/yarn/nm-local-dir/usercache/knu/appcache/application_1491995822063_0001/container_1491995822063_0001_01_000003/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000003 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 39313 attempt_1491995822063_0001_m_000001_0 3 1>/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000003/stdout 2>/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000003/stderr  

Container killed on request. Exit code is 143

Container exited with a non-zero exit code 143

17/04/12 21:42:01 INFO mapreduce.Job: Task Id : attempt_1491995822063_0001_m_000000_0, Status : FAILED

Container [pid=12054,containerID=container_1491995822063_0001_01_000002] is running beyond virtual memory limits. Current usage: 107.3 MB of 1 GB physical memory used; 2.1 GB of 2.1 GB virtual memory used. Killing container.

Dump of the process-tree for container_1491995822063_0001_01_000002 :

    |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE

    |- 12072 12054 12054 12054 (java) 204 9 2154971136 27169 /home/knu/hadoop/jdk1.8.0_121/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx400M -Djava.io.tmpdir=/home/knu/hadoop/hadoop-2.7.3/data/yarn/nm-local-dir/usercache/knu/appcache/application_1491995822063_0001/container_1491995822063_0001_01_000002/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 39313 attempt_1491995822063_0001_m_000000_0 2 

    |- 12054 12052 12054 12054 (bash) 0 0 115847168 302 /bin/bash -c /home/knu/hadoop/jdk1.8.0_121/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN  -Xmx400M -Djava.io.tmpdir=/home/knu/hadoop/hadoop-2.7.3/data/yarn/nm-local-dir/usercache/knu/appcache/application_1491995822063_0001/container_1491995822063_0001_01_000002/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 39313 attempt_1491995822063_0001_m_000000_0 2 1>/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000002/stdout 2>/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000002/stderr  

Container killed on request. Exit code is 143

Container exited with a non-zero exit code 143

17/04/12 21:42:07 INFO mapreduce.Job: Task Id : attempt_1491995822063_0001_m_000001_1, Status : FAILED

Container [pid=12143,containerID=container_1491995822063_0001_01_000004] is running beyond virtual memory limits. Current usage: 105.0 MB of 1 GB physical memory used; 2.1 GB of 2.1 GB virtual memory used. Killing container.

Dump of the process-tree for container_1491995822063_0001_01_000004 :

    |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE

    |- 12151 12143 12143 12143 (java) 177 5 2153918464 26571 /home/knu/hadoop/jdk1.8.0_121/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx400M -Djava.io.tmpdir=/home/knu/hadoop/hadoop-2.7.3/data/yarn/nm-local-dir/usercache/knu/appcache/application_1491995822063_0001/container_1491995822063_0001_01_000004/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000004 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 39313 attempt_1491995822063_0001_m_000001_1 4 

    |- 12143 12142 12143 12143 (bash) 0 0 115847168 303 /bin/bash -c /home/knu/hadoop/jdk1.8.0_121/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN  -Xmx400M -Djava.io.tmpdir=/home/knu/hadoop/hadoop-2.7.3/data/yarn/nm-local-dir/usercache/knu/appcache/application_1491995822063_0001/container_1491995822063_0001_01_000004/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000004 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 39313 attempt_1491995822063_0001_m_000001_1 4 1>/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000004/stdout 2>/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000004/stderr  

Container killed on request. Exit code is 143

Container exited with a non-zero exit code 143

17/04/12 21:42:10 INFO mapreduce.Job: Task Id : attempt_1491995822063_0001_m_000000_1, Status : FAILED

Container [pid=12164,containerID=container_1491995822063_0001_01_000005] is running beyond virtual memory limits. Current usage: 130.7 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used. Killing container.

Dump of the process-tree for container_1491995822063_0001_01_000005 :

    |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE

    |- 12172 12164 12164 12164 (java) 293 9 2175307776 32535 /home/knu/hadoop/jdk1.8.0_121/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx400M -Djava.io.tmpdir=/home/knu/hadoop/hadoop-2.7.3/data/yarn/nm-local-dir/usercache/knu/appcache/application_1491995822063_0001/container_1491995822063_0001_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 39313 attempt_1491995822063_0001_m_000000_1 5 

    |- 12164 12163 12164 12164 (bash) 0 0 115847168 302 /bin/bash -c /home/knu/hadoop/jdk1.8.0_121/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN  -Xmx400M -Djava.io.tmpdir=/home/knu/hadoop/hadoop-2.7.3/data/yarn/nm-local-dir/usercache/knu/appcache/application_1491995822063_0001/container_1491995822063_0001_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 39313 attempt_1491995822063_0001_m_000000_1 5 1>/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000005/stdout 2>/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000005/stderr  

    |- 12224 12172 12164 12164 (R) 0 0 116293632 469 /bin/sh /usr/lib64/R/bin/R --slave --no-restore --vanilla --file=./rmr-streaming-map2d5212712621 

    |- 12228 12224 12164 12164 (R) 0 0 116293632 158 /bin/sh /usr/lib64/R/bin/R --slave --no-restore --vanilla --file=./rmr-streaming-map2d5212712621 

Container killed on request. Exit code is 143

Container exited with a non-zero exit code 143

17/04/12 21:42:16 INFO mapreduce.Job: Task Id : attempt_1491995822063_0001_m_000001_2, Status : FAILED

Container [pid=12254,containerID=container_1491995822063_0001_01_000007] is running beyond virtual memory limits. Current usage: 160.6 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used. Killing container.

Dump of the process-tree for container_1491995822063_0001_01_000007 :

    |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE

    |- 12319 12262 12254 12254 (R) 10 2 268447744 8256 /usr/lib64/R/bin/exec/R --slave --no-restore --vanilla --file=./rmr-streaming-map2d5212712621 

    |- 12262 12254 12254 12254 (java) 295 9 2175307776 32549 /home/knu/hadoop/jdk1.8.0_121/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx400M -Djava.io.tmpdir=/home/knu/hadoop/hadoop-2.7.3/data/yarn/nm-local-dir/usercache/knu/appcache/application_1491995822063_0001/container_1491995822063_0001_01_000007/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000007 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 39313 attempt_1491995822063_0001_m_000001_2 7 

    |- 12254 12253 12254 12254 (bash) 0 0 115847168 302 /bin/bash -c /home/knu/hadoop/jdk1.8.0_121/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN  -Xmx400M -Djava.io.tmpdir=/home/knu/hadoop/hadoop-2.7.3/data/yarn/nm-local-dir/usercache/knu/appcache/application_1491995822063_0001/container_1491995822063_0001_01_000007/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000007 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 39313 attempt_1491995822063_0001_m_000001_2 7 1>/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000007/stdout 2>/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000007/stderr  

Container killed on request. Exit code is 143

Container exited with a non-zero exit code 143

17/04/12 21:42:16 INFO mapreduce.Job: Task Id : attempt_1491995822063_0001_m_000000_2, Status : FAILED

Container [pid=12280,containerID=container_1491995822063_0001_01_000008] is running beyond virtual memory limits. Current usage: 92.1 MB of 1 GB physical memory used; 2.1 GB of 2.1 GB virtual memory used. Killing container.

Dump of the process-tree for container_1491995822063_0001_01_000008 :

    |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE

    |- 12280 12279 12280 12280 (bash) 0 0 115847168 303 /bin/bash -c /home/knu/hadoop/jdk1.8.0_121/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN  -Xmx400M -Djava.io.tmpdir=/home/knu/hadoop/hadoop-2.7.3/data/yarn/nm-local-dir/usercache/knu/appcache/application_1491995822063_0001/container_1491995822063_0001_01_000008/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000008 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 39313 attempt_1491995822063_0001_m_000000_2 8 1>/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000008/stdout 2>/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000008/stderr  

    |- 12289 12280 12280 12280 (java) 137 5 2142965760 23280 /home/knu/hadoop/jdk1.8.0_121/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx400M -Djava.io.tmpdir=/home/knu/hadoop/hadoop-2.7.3/data/yarn/nm-local-dir/usercache/knu/appcache/application_1491995822063_0001/container_1491995822063_0001_01_000008/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/knu/hadoop/hadoop-2.7.3/logs/userlogs/application_1491995822063_0001/container_1491995822063_0001_01_000008 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 39313 attempt_1491995822063_0001_m_000000_2 8 

Container killed on request. Exit code is 143

Container exited with a non-zero exit code 143

17/04/12 21:42:23 INFO mapreduce.Job:  map 100% reduce 0%

17/04/12 21:42:23 INFO mapreduce.Job: Job job_1491995822063_0001 failed with state FAILED due to: Task failed task_1491995822063_0001_m_000001

Job failed as tasks failed. failedMaps:1 failedReduces:0

17/04/12 21:42:23 INFO mapreduce.Job: Counters: 13

    Job Counters 

        Failed map tasks=7

        Killed map tasks=1

        Launched map tasks=8

        Other local map tasks=6

        Data-local map tasks=2

        Total time spent by all maps in occupied slots (ms)=39157

        Total time spent by all reduces in occupied slots (ms)=0

        Total time spent by all map tasks (ms)=39157

        Total vcore-milliseconds taken by all map tasks=39157

        Total megabyte-milliseconds taken by all map tasks=40096768

    Map-Reduce Framework

        CPU time spent (ms)=0

        Physical memory (bytes) snapshot=0

        Virtual memory (bytes) snapshot=0

17/04/12 21:42:23 ERROR streaming.StreamJob: Job not successful!

Streaming Command Failed!

Error in mr(map = map, reduce = reduce, combine = combine, vectorized.reduce,  : 

  hadoop streaming failed with error code 1
hadoopmapreducerhadoop

暂无答案！

目前还没有任何答案，快来回答吧！

我来回答

相关问题

    查看更多
    微信公众号

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com