2.7.1集群上

9rnv2umw  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(256)

我已经在多节点集群(1个namenode和4个datanode)上成功运行了hadoop2.7.1。但是,当我运行mapreduce作业(来自hadoop网站的wordcount示例)时,它总是停留在这一点上。

[~@~ hadoop-2.7.1]$ bin/hadoop jar WordCount.jar WordCount /user/inputdata/ /user/outputdata
15/09/30 17:54:56 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/09/30 17:54:57 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
15/09/30 17:54:58 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
15/09/30 17:54:59 INFO input.FileInputFormat: Total input paths to process : 1
15/09/30 17:55:00 INFO mapreduce.JobSubmitter: number of splits:1
15/09/30 17:55:00 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1443606819488_0002
15/09/30 17:55:00 INFO impl.YarnClientImpl: Submitted application application_1443606819488_0002
15/09/30 17:55:00 INFO mapreduce.Job: The url to track the job: http://~~~~:8088/proxy/application_1443606819488_0002/
15/09/30 17:55:00 INFO mapreduce.Job: Running job: job_1443606819488_0002

我必须为指定内存吗 yarn ?
注意:datanode硬件非常旧(每个都有1gb ram)。
谢谢你的帮助。谢谢您。

mznpcxlj

mznpcxlj1#

数据节点内存(1gb)非常有限,至少需要准备一个容器来运行mapper/reducer/am。
您可以尝试在中降低以下容器内存分配值 yarn-site.xml 以获得在其上创建的容器。

yarn.scheduler.minimum-allocation-mb
yarn.scheduler.maximum-allocation-mb

还要尽量减少作业配置中的以下属性值,

mapreduce.map.memory.mb
mapreduce.reduce.memory.mb
mapreduce.map.java.opts
mapreduce.reduce.java.opts

相关问题