我对大数据和spark非常陌生,下面是我如何尝试获得spark课程的
SparkConf conf = new SparkConf().setMaster("local").setAppName("SaavnAnalyticsProject");
sparkSession = SparkSession.builder().config(conf).getOrCreate();
这就是我犯的错误
使用spark的默认log4j配置文件:org/apache/spark/log4j-defaults.properties 20/02/03 02:29:40 info sparkcontext:在线程“main”java.lang.nosuchmethoderror:com.google.common.base.joiner.on(c)lcom/google/common/base/joiner中运行spark版本2.3.0异常;在org.apache.hadoop.metrics2.lib.uniquenames。java:44)在org.apache.hadoop.metrics2.lib.defaultmetricssystem.(defaultmetricssystem。java:41)在org.apache.hadoop.metrics2.lib.defaultmetricssystem.(defaultmetricssystem。java:36)在org.apache.hadoop.security.usergroupinformation$ugimetrics.create(usergroupinformation。java:120)在org.apache.hadoop.security.usergroupinformation.(usergroupinformation。java:236)在org.apache.spark.util.utils$$anonfun$getcurrentusername$1.apply(utils。scala:2464)在org.apache.spark.util.utils$$anonfun$getcurrentusername$1.apply(utils。scala:2464)在scala.option.getorelse(option。scala:121)在org.apache.spark.util.utils$.getcurrentusername(utils。scala:2464)在org.apache.spark.sparkcontext。scala:292)在org.apache.spark.sparkcontext$.getorcreate(sparkcontext。scala:2486)在org.apache.spark.sql.sparksession$builder$$anonfun$7.apply(sparksession。scala:930)在org.apache.spark.sql.sparksession$builder$$anonfun$7.apply(sparksession。scala:921)在scala.option.getorelse(option。scala:121)在org.apache.spark.sql.sparksession$builder.getorcreate(sparksession。scala:921)在saavnalytics.saavnalyticsmain.main(saavnalyticsmain。java:55)
这是我的pom.xml
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.target>1.8</maven.compiler.target>
<maven.compiler.source>1.8</maven.compiler.source>
</properties>
<dependencies>
<dependency>
<!-- Apache Spark main library -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<version>2.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.11</artifactId>
<version>2.3.0</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-aws</artifactId>
<version>2.7.1</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>2.6.0</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.6.0</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk</artifactId>
<version>1.7.4</version>
</dependency>
</dependencies>
<build>
<plugins>
<!-- Maven Shade Plugin -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<executions>
<!-- Run shade goal on package phase -->
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>SaavnAnalytics.SaavnAnalyticsMain</mainClass>
</transformer>
</transformers>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
</plugins>
我已经显式地添加了google-collect-0.5.jar和com.google.collections.jar。
知道我哪里出错了吗?
1条答案
按热度按时间ncgqoxb01#
尝试修改pom.xml plugins部分: