如何解析securitymanager:已禁用身份验证;spark distribution cluster异常

vfh0ocws  于 2021-06-27  发布在  Java
关注(0)|答案(0)|浏览(327)

我在linux(ubuntu)环境下创建了spark分发集群来运行spark提交作业,我有3台机器,如:m1、s1和s2。这里m1是主机s1,s2是从机。
在master spark和slave spark目录下../spark/conf/spark-defaults.conf,spark-env.sh我已经配置好了所有的东西。
spark-env.sh公司

export SPARK_MASTER_HOST=198.77.8.7
export SPARK_WORKER_HOST=198.77.8.12
export SPARK_WORKER_HOST=198.77.8.18
export SPARK_LOCAL_IP=198.77.8.7
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export SPARK_WORKER_CORES=8

spark-defaults.conf格式

spark.master yarn
spark.executor.memory 1g
spark.driver.memory 1g
spark.yarn.am.memory 1g

当我使用客户机模式运行命令时,它正在工作,命令是:
bin/spark submit--master local[*]--deploy mode client--class com.x.y.app--executor memory 15g--executor cores 2--driver memory 15g/home/user/workspace/test.jar
上面的命令正在运行,我正在获取输出文件,但是当我尝试在集群模式下运行时,它给了我一个异常,命令是。
bin/spark提交--主spark://198.47.12.8:7077--部署模式集群--类com.x.y.app--执行器内存15g--执行器内核2--驱动程序内存15g/home/user/workspace/test.jar
我得到了这样的例外:

21/01/08 16:51:51 INFO SecurityManager: Changing modify acls groups to: 
21/01/08 16:51:51 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(sys92); groups with view permissions: Set(); users  with modify permissions: Set(sys92); groups with modify permissions: Set()
21/01/08 16:51:51 WARN Utils: Service 'Driver' could not bind on a random free port. You may check whether configuring an appropriate binding address.
21/01/08 16:51:51 WARN Utils: Service 'Driver' could not bind on a random free port. You may check whether configuring an appropriate binding address.
21/01/08 16:51:51 WARN Utils: Service 'Driver' could not bind on a random free port. You may check whether configuring an appropriate binding address.
21/01/08 16:51:51 WARN Utils: Service 'Driver' could not bind on a random free port. You may check whether configuring an appropriate binding address.
21/01/08 16:51:51 WARN Utils: Service 'Driver' could not bind on a random free port. You may check whether configuring an appropriate binding address.
21/01/08 16:51:51 WARN Utils: Service 'Driver' could not bind on a random free port. You may check whether configuring an appropriate binding address.
21/01/08 16:51:51 WARN Utils: Service 'Driver' could not bind on a random free port. You may check whether configuring an appropriate binding address.
21/01/08 16:51:51 WARN Utils: Service 'Driver' could not bind on a random free port. You may check whether configuring an appropriate binding address.
21/01/08 16:51:51 WARN Utils: Service 'Driver' could not bind on a random free port. You may check whether configuring an appropriate binding address.
21/01/08 16:51:51 WARN Utils: Service 'Driver' could not bind on a random free port. You may check whether configuring an appropriate binding address.
21/01/08 16:51:51 WARN Utils: Service 'Driver' could not bind on a random free port. You may check whether configuring an appropriate binding address.
21/01/08 16:51:51 WARN Utils: Service 'Driver' could not bind on a random free port. You may check whether configuring an appropriate binding address.
21/01/08 16:51:51 WARN Utils: Service 'Driver' could not bind on a random free port. You may check whether configuring an appropriate binding address.
21/01/08 16:51:51 WARN Utils: Service 'Driver' could not bind on a random free port. You may check whether configuring an appropriate binding address.
21/01/08 16:51:51 WARN Utils: Service 'Driver' could not bind on a random free port. You may check whether configuring an appropriate binding address.
21/01/08 16:51:51 WARN Utils: Service 'Driver' could not bind on a random free port. You may check whether configuring an appropriate binding address.
Exception in thread "main" java.net.BindException: Cannot assign requested address: Service 'Driver' failed after 16 retries (on a random free port)! Consider explicitly setting the appropriate binding address for the service 'Driver' (for example spark.driver.bindAddress for SparkDriver) to the correct binding address.
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:461)
    at sun.nio.ch.Net.bind(Net.java:453)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:222)
    at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:134)
    at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:550)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1334)
    at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:506)
    at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:491)
    at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:973)
    at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:248)
    at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:356)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)

我已经检查了所有3个系统/etc/hosts文件,我认为这不是问题。如果需要任何额外的信息或配置文件,请告诉我。
主etc/主机


# 127.0.0.1  localhost

# 127.0.1.1  mater1

198.77.8.7   mater1
198.77.8.12  slave1
198.77.8.18  slave2

salve1等/主机


# 127.0.0.1  localhost

# 127.0.1.1  slave1

198.77.8.7   mater1
198.77.8.12  slave1
198.77.8.18  slave2

从属2等/主机


# 127.0.0.1  localhost

# 127.0.1.1  slave2

198.77.8.7   mater1
198.77.8.12  slave1
198.77.8.18  slave2

那我怎么才能找出根本原因呢。有人能帮我找到问题吗。
谢谢,
比马尔。

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题