如何使用Java8和Spark2.3.0创建sparksession

w3nuxt5m  于 2021-05-27  发布在  Hadoop
关注(0)|答案(1)|浏览(375)

我对大数据和spark非常陌生,下面是我如何尝试获得spark课程的

SparkConf conf = new SparkConf().setMaster("local").setAppName("SaavnAnalyticsProject");
sparkSession = SparkSession.builder().config(conf).getOrCreate();

这就是我犯的错误
使用spark的默认log4j配置文件:org/apache/spark/log4j-defaults.properties 20/02/03 02:29:40 info sparkcontext:在线程“main”java.lang.nosuchmethoderror:com.google.common.base.joiner.on(c)lcom/google/common/base/joiner中运行spark版本2.3.0异常;在org.apache.hadoop.metrics2.lib.uniquenames。java:44)在org.apache.hadoop.metrics2.lib.defaultmetricssystem.(defaultmetricssystem。java:41)在org.apache.hadoop.metrics2.lib.defaultmetricssystem.(defaultmetricssystem。java:36)在org.apache.hadoop.security.usergroupinformation$ugimetrics.create(usergroupinformation。java:120)在org.apache.hadoop.security.usergroupinformation.(usergroupinformation。java:236)在org.apache.spark.util.utils$$anonfun$getcurrentusername$1.apply(utils。scala:2464)在org.apache.spark.util.utils$$anonfun$getcurrentusername$1.apply(utils。scala:2464)在scala.option.getorelse(option。scala:121)在org.apache.spark.util.utils$.getcurrentusername(utils。scala:2464)在org.apache.spark.sparkcontext。scala:292)在org.apache.spark.sparkcontext$.getorcreate(sparkcontext。scala:2486)在org.apache.spark.sql.sparksession$builder$$anonfun$7.apply(sparksession。scala:930)在org.apache.spark.sql.sparksession$builder$$anonfun$7.apply(sparksession。scala:921)在scala.option.getorelse(option。scala:121)在org.apache.spark.sql.sparksession$builder.getorcreate(sparksession。scala:921)在saavnalytics.saavnalyticsmain.main(saavnalyticsmain。java:55)
这是我的pom.xml

<properties>
 <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <maven.compiler.target>1.8</maven.compiler.target>
        <maven.compiler.source>1.8</maven.compiler.source>
    </properties>
  <dependencies>
    <dependency>
        <!-- Apache Spark main library -->  
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <version>2.3.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.11</artifactId>
        <version>2.3.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming_2.11</artifactId>
        <version>2.3.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-mllib_2.11</artifactId>
        <version>2.3.0</version>
    </dependency>
        <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>

    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-aws</artifactId>
        <version>2.7.1</version>
    </dependency>
    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-annotations</artifactId>
        <version>2.6.0</version>
    </dependency>
    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-databind</artifactId>
        <version>2.6.0</version>
    </dependency>
    <dependency>
      <groupId>com.amazonaws</groupId>
      <artifactId>aws-java-sdk</artifactId>
      <version>1.7.4</version>
    </dependency>
    </dependencies>
    <build>
   <plugins>
        <!-- Maven Shade Plugin -->
        <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-shade-plugin</artifactId>
          <version>2.3</version>
          <executions>
             <!-- Run shade goal on package phase -->
            <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
            <configuration>
             <transformers>
                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                    <mainClass>SaavnAnalytics.SaavnAnalyticsMain</mainClass>
            </transformer>
              </transformers>
            <filters>             
            <filter>               
            <artifact>*:*</artifact>               
            <excludes>                 
            <exclude>META-INF/*.SF</exclude>                 
            <exclude>META-INF/*.DSA</exclude>                 
            <exclude>META-INF/*.RSA</exclude>               
            </excludes>             
            </filter>           
            </filters>         
            </configuration>
            </execution>
          </executions>
        </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
        </plugins>

我已经显式地添加了google-collect-0.5.jar和com.google.collections.jar。
知道我哪里出错了吗?

ncgqoxb0

ncgqoxb01#

尝试修改pom.xml plugins部分:

<plugins>
    <!-- Maven Shade Plugin -->
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>2.3</version>
        <executions>
            <!-- Run shade goal on package phase -->
            <execution>
                <phase>package</phase>
                <goals>
                    <goal>shade</goal>
                </goals>
                <configuration>
                    <relocations>
                        <relocation>
                            <pattern>com.google.common</pattern>
                            <shadedPattern>shade.com.google.common</shadedPattern>
                        </relocation>
                    </relocations>
                    <transformers>
                        <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                            <mainClass>SaavnAnalytics.SaavnAnalyticsMain</mainClass>
                        </transformer>
                    </transformers>
                    <filters>
                        <filter>
                            <artifact>*:*</artifact>
                            <excludes>
                                <exclude>META-INF/*.SF</exclude>
                                <exclude>META-INF/*.DSA</exclude>
                                <exclude>META-INF/*.RSA</exclude>
                            </excludes>
                        </filter>
                    </filters>
                </configuration>
            </execution>
        </executions>
    </plugin>
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <configuration>
            <source>1.8</source>
            <target>1.8</target>
        </configuration>
    </plugin>
</plugins>

相关问题