在Yarn中启动spark应用程序时出错

qltillow  于 2021-05-30  发布在  Hadoop
关注(0)|答案(0)|浏览(403)

我试图运行我的Spark字计数程序使用Yarn作为集群管理器。我使用的是hadoop2.6&yarn被配置为以伪分布式模式运行。申请提交失败

spark-submit --master yarn --class sbook.helloworld.WordCount 
target/scala-2.11/sparkbookapp_2.11-1.0.jar 
src/main/resources/data.txt  output

下面给出的是我能够从节点管理器中检索到的错误(使用级别 checkout 日志) ERROR ). 看起来这似乎是集装箱分配的问题,但找不到原因

INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]
INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1434487429379_0008_000002
INFO spark.SecurityManager: Changing view acls to: mountain
INFO spark.SecurityManager: Changing modify acls to: mountain
INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(mountain); users with modify permissions: Set(mountain)
WARN util.Utils: Your hostname, mountain resolves to a loopback address: 127.0.1.1; using 10.0.0.6 instead (on interface wlan0)
WARN util.Utils: Set SPARK_LOCAL_IP if you need to bind to another address
INFO slf4j.Slf4jLogger: Slf4jLogger started
INFO Remoting: Starting remoting
INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkYarnAM@10.0.0.6:42711]
INFO util.Utils: Successfully started service 'sparkYarnAM' on port 42711.
INFO yarn.ApplicationMaster: Waiting for Spark driver to be reachable.
INFO yarn.ApplicationMaster: Driver now available: 10.0.0.6:34231
INFO yarn.ApplicationMaster: Listen to driver: akka.tcp://sparkDriver@10.0.0.6:34231/user/YarnScheduler
INFO yarn.ApplicationMaster: Add WebUI Filter. AddWebUIFilter(org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter,Map(PROXY_HOSTS -> mountain, PROXY_URI_BASES -> http://mountain:8088/proxy/application_1434487429379_0008),/proxy/application_1434487429379_0008)
INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8030
    15/06/16 16:11:02 INFO yarn.YarnRMClient: Registering the ApplicationMaster
INFO yarn.YarnAllocator: Will request 2 executor containers, each with 1 cores and 1408 MB memory including 384 MB overhead
INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:1408, vCores:1>)
INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:1408, vCores:1>)
INFO yarn.ApplicationMaster: Started progress reporter thread - sleep time : 5000
ERROR yarn.ApplicationMaster: RECEIVED SIGNAL 15: SIGTERM
INFO yarn.ApplicationMaster: Final app status: UNDEFINED, exitCode: 0, (reason: Shutdown hook called before final status was reported.)
INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with UNDEFINED (diag message: Shutdown hook called before final status was reported.)
INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered.
INFO yarn.ApplicationMaster: Deleting staging directory .sparkStaging/application_1434487429379_0008

我也尝试过将executor的大小改为1g英寸 spark-submit 脚本。但这也没有帮助,节点管理器仍在尝试创建一个大小为1408MB的执行器。

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题