无法连接到使用spring boot在远程服务器上设置的远程独立apache spark

doinxwow  于 2021-05-26  发布在  Spark
关注(0)|答案(0)|浏览(294)

我想连接到apachespark,它是从本地机器上运行的spring启动应用程序在远程服务器上设置的。我在网上看到了一些示例,但都在本地系统上配置了spark,并在spark.home中提供了可安装的路径。
我想要一个类似的设置,但不是Spark.home指向本地系统,它应该连接到一个远程机器。任何帮助都将不胜感激。
下面是代码

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaSparkContext;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;

@SpringBootApplication

public class SpringSparkApplication {

    @Value("${spark.app.name}")
    private String appName;
    @Value("${spark.master}")
    private String masterUri;

    @Bean
    public SparkConf conf() {
        return new SparkConf().setAppName(appName).setMaster(masterUri);
    }

    @Bean
    public JavaSparkContext sc() {
        return new JavaSparkContext(conf());
    }

    public static void main(String[] args) {
        SpringApplication.run(SpringSparkApplication.class, args);
    }

}

这些属性是:
spark.app.name=ayush spark.master=spark://192.x.x.50:7077
获取错误为:
2020-10-14 16:07:26.098 info 25028---[main]org.apache.spark.ui.sparkui:将sparkui绑定到0.0.0.0,开始于http://amaheshwari-pc:4040 2020-10-14 16:07:26.187信息25028---[er-threadpool-0]s.d.c.standaloneappclient$clientendpoint:连接到主服务器spark://192.x.x.50:7077... 2020-10-14 16:07:26.449 info 25028---[pc-connection-0]o.a.s.n.client.transportclientfactory:251毫秒后成功创建到/192.x.x.50:7077的连接(引导过程中花费0毫秒)2020-10-14 16:07:46.205 info 25028---[er-threadpool-0]s.d.c.standaloneappclient$clientendpoint:连接到主机spark://192.x.x.50:7077... 2020-10-14 16:08:06.207信息25028---[er-threadpool-0]s.d.c.standaloneappclient$clientendpoint:连接到主服务器spark://192.x.x.50:7077... 2020-10-14 16:08:26.221错误25028---[重试线程上]o.a.s.s.c.StandalonesSchedulerBackend:应用程序已被终止。原因:所有的主人都没有React!放弃。2020-10-14 16:08:26.222警告25028---[main]o.a.s.s.c.StandalonesSchedulerBackend:应用程序id尚未初始化。2020-10-14 16:08:26.230信息25028---[p-spark-context]o.s.jetty.server.abstractconnector:已停止spark@646427f7{http/1.1,[http/1.1]}{0.0.0.0:4040}2020-10-14 16:08:26.231 info 25028---[p-spark-context]org.apache.spark.ui.sparkui:spark web ui停止于http://amaheshwari-pc:4040 2020-10-14 16:08:26.237 info 25028---[p-spark-context]o.a.s.s.c.standaloneschedulerbackend:关闭所有执行器2020-10-14 16:08:26.238 info 25028---[main]org.apache.spark.util.utils:已在端口56322上成功启动服务“org.apache.spark.network.netty.nettyblocktransferservice”。2020-10-14 16:08:26.239 info 25028---[main]o.a.s.n.netty.nettyblocktransferservice:在amaheshwari pc上创建的服务器:56322 2020-10-14 16:08:26.240 info 25028---[main]org.apache.spark.storage.blockmanager:对块复制策略2020-10-14使用org.apache.spark.storage.randomblockreplicationpolicy[er-event-loop-8]segrainedschedulerbackend$driverendpoint:要求每个执行者关闭2020-10-14 16:08:26.243 warn 25028---[r-event-loop-11]s.d.c.standaloneappclient$clientendpoint:删除unregisterapplication(null),因为尚未连接到主机2020-10-14 16:08:26.249 info 25028---[er-event-loop-5]o、 a.s.mapoutputtrackermasterendpoint:mapoutputtrackermasterendpoint已停止!2020-10-14 16:08:26.256 info 25028---[p-spark-context]o.a.spark.storage.memory.memorystore:memorystore清除的内存存储2020-10-14 16:08:26.257 info 25028---[p-spark-context]org.apache.spark.storage.blockmanager:blockmanager停止2020-10-14 16:08:26.262 info 25028---[main]o.a.spark.storage.blockmanager主程序:注册blockmanagerblockmanagerid(驱动程序,amaheshwari-pc.xsolutions.com,56322,无)2020-10-14 16:08:26.262 info 25028---[p-spark-context]o.a.spark.storage.blockmanagermaster:blockmanagermaster stopped 2020-10-14 16:08:26.263 warn 25028---[p-spark-context]org.apache.spark.metrics.metricssystem:停止未运行2020-10-14 16:08:26.263 error 25028---[main]org.apache.spark.sparkcontext:初始化sparkcontext时出错。
java.lang.nullpointerexception:org.apache.spark.storage.blockmanagermaster.registerblockmanager(blockmanagermaster)上的null。scala:64)~[Spark芯2.12-2.4.0。jar:2.4.0]在org.apache.spark.storage.blockmanager.initialize(blockmanager。scala:252)~[Spark芯2.12-2.4.0。jar:2.4.0]在org.apache.spark.sparkcontext。scala:510) ~[Spark芯2.12-2.4.0。jar:2.4.0]在org.apache.spark.api.java.javasparkcontext。scala:58)[Spark芯2.12-2.4.0。jar:2.4.0]在com.x.springspark.springsparkapplication.sc(springsparkapplication。java:25)[classes/:na]位于com.x.springspark.springsparkapplication$$enhancerbyspringcglib$$5abe7298.cglib$sc$1()[classes/:na]位于springspark.springsparkapplication$$enhancerbyspringcglib$$5abe7298$$fastclassbysspringcglib$$81e62437.invoke()[classes/:na]位于org.springframework.cglib.proxy.methodproxy.invokesuper(methodproxy)。java:244)[Spring芯-5.2.9.释放。jar:5.2.9.release]在org.springframework.context.annotation.configurationclassenhancer$beanmethodinterceptor.intercept(配置类增强器)。java:331)[spring-context-5.2.9.发布。jar:5.2.9.release]在com.x.springspark.springsparkapplication$$enhancerbyspringcglib$$5abe7298.sc()[classes/:na]的sun.reflect.nativemethodaccessorimpl.invoke0(本机方法)~[na:1.8.0\u 261]在sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl。java:62)~[na:1.8.0\u 261]在sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl。java:43)~[na:1.8.0\u 261]位于java.lang.reflect.method.invoke(method。java:498)~(na:1.8.0μ261)在org.springframework.beans.factory.support.simpleinstationstrategy.instantiate(simpleinstationstrategy)。java:154)[spring-beans-5.2.9.发布。jar:5.2.9.release]在org.springframework.beans.factory.support.constructorresolver.instantiate(constructorresolver。java:650)[spring-beans-5.2.9.发布。jar:5.2.9.release]在org.springframework.beans.factory.support.constructorresolver.instantialEusingFactoryMethod(constructorresolver。java:483)[spring-beans-5.2.9.发布。jar:5.2.9.release]位于org.springframework.beans.factory.support.abstractautowirecapablebeanfactory.instantialEUsingFactoryMethod(abstractautowirecapablebeanfactory)。java:1336) [spring-beans-5.2.9.发布。jar:5.2.9.release]位于org.springframework.beans.factory.support.abstractautowirecapablebeanfactory.createbeaninstance(abstractautowirecapablebeanfactory)。java:1176)[spring-beans-5.2.9.发布。jar:5.2.9.release]在org.springframework.beans.factory.support.abstractautowirecapablebeanfactory.docreatebean(abstractautowirecapablebeanfactory)。java:556)[spring-beans-5.2.9.发布。jar:5.2.9.release]位于org.springframework.beans.factory.support.abstractautowirecapablebeanfactory.createbean(abstractautowirecapablebeanfactory)。java:516) [spring-beans-5.2.9.发布。jar:5.2.9.release]在org.springframework.beans.factory.support.abstractbeanfactory.lambda$dogetbean$0(abstractbeanfactory)。java:324)[spring-beans-5.2.9.发布。jar:5.2.9.release]位于org.springframework.beans.factory.support.defaultsingletonbeanregistry.getsingleton(defaultsingletonbeanregistry)。java:234) ~[spring-beans-5.2.9.发布。jar:5.2.9.release]位于org.springframework.beans.factory.support.abstractbeanfactory.dogetbean(abstractbeanfactory.com)。java:322)[spring-beans-5.2.9.发布。jar:5.2.9.release]在org.springframework.beans.factory.support.abstractbeanfactory.getbean(abstractbeanfactory。java:202)[spring-beans-5.2.9.发布。jar:5.2.9.release] 位于org.springframework.beans.factory.support.defaultlistablebeanfactory.PreInstanceSingletons(defaultlistablebeanfactory)。java:897)~[spring-beans-5.2.9.发布。jar:5.2.9.release]位于org.springframework.context.support.abstractapplicationcontext.finishbeanfactoryinitialization(abstractapplicationcontext)。java:879) ~[spring-context-5.2.9.发布。jar:5.2.9.release]在org.springframework.context.support.abstractapplicationcontext.refresh(abstractapplicationcontext。java:551)~[spring-context-5.2.9.发布。jar:5.2.9.release]在org.springframework.boot.web.servlet.context.servletwebserverapplicationcontext.refresh(servletwebserverapplicationcontext。java:143) ~[spring-boot-2.3.4.释放。jar:2.3.4.release]在org.springframework.boot.springapplication.refresh(springapplication。java:758)~[spring-boot-2.3.4.释放。jar:2.3.4.release]在org.springframework.boot.springapplication.refresh(springapplication。java:750)~[spring-boot-2.3.4.释放。jar:2.3.4.release]在org.springframework.boot.springapplication.refreshcontext(springapplication。java:397)~[spring-boot-2.3.4.释放。jar:2.3.4.release]在org.springframework.boot.springapplication.run(springapplication。java:315)~[spring-boot-2.3.4.释放。jar:2.3.4.release]在org.springframework.boot.springapplication.run(springapplication。java:1237) ~[spring-boot-2.3.4.释放。jar:2.3.4.release]在org.springframework.boot.springapplication.run(springapplication。java:1226)~[spring-boot-2.3.4.释放。jar:2.3.4.release]在com.x.springspark.springsparkapplication.main(springsparkapplication。java:29)[类/:an]
2020-10-14 16:08:26.263信息25028---[main]org.apache.spark.sparkcontext:sparkcontext已经停止。2020-10-14 16:08:26.265信息25028---[er-event-loop-7]协调器$outputcommitocordinatorendpoint:outputcommitocordinator停止!2020-10-14 16:08:26.265警告25028---[main]configservletwebserverapplicationcontext:上下文初始化期间遇到异常-取消刷新尝试:org.springframework.beans.factory.beancreationexception:创建名为“sc”的bean时出错,该bean在com.x.springspark.springsparkapplication:通过工厂示例化bean方法失败;嵌套异常为org.springframework.beans.beaninstantiationexception:未能示例化[org.apache.spark.api.java.javasparkcontext]:工厂方法“sc”引发异常;嵌套异常为java.lang.nullpointerexception 2020-10-14 16:08:26.268 info 25028---[main]o.apache.catalina.core.standardservice:停止服务[tomcat]2020-10-14 16:08:26.271 info 25028---[p-spark-context]org.apache.spark.sparkcontext:成功停止sparkcontext 2020-10-14 16:08:26.278 info 25028---[main]条件评估报告日志侦听器:
启动applicationcontext时出错。要显示条件报告,请在启用“调试”的情况下重新运行应用程序。2020-10-14 16:08:26.285错误25028---[main]o.s.boot.spr

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题