尝试在本地模型中提交spark应用程序,出现以下错误“无法从jar加载主类”

fkvaft9z  于 2021-05-18  发布在  Spark
关注(0)|答案(0)|浏览(197)

我试图提交一个Spark在我的本地申请,我得到下面的错误。

Exception in thread "main" org.apache.spark.SparkException: Cannot load main class from JAR file:        at org.apache.spark.deploy.SparkSubmitArguments.error(SparkSubmitArguments.scala:657)
        at org.apache.spark.deploy.SparkSubmitArguments.loadEnvironmentArguments(SparkSubmitArguments.scala:221)
        at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:116)
        at org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$1.<init>(SparkSubmit.scala:907)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.parseArguments(SparkSubmit.scala:907)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:81)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

下面是我用来提交spark申请的命令
spark提交单词\u count.py
我不知道我到底错过了什么,任何帮助都将不胜感激

from pyspark.sql import SparkSession
from pyspark.sql.functions import explode
from pyspark.sql.functions import split

def main():
    sparkSession = SparkSession.builder.appName("Word Count").getOrCreate()
    sparkSession.sparkContext.setLogLevel("ERROR")
    readStream = sparkSession.readStream.format('text').load(path)

    print("-------------------------------------------------")  
    print("Streaming source ready: ", readStream.isStreaming)
    readStream.printSchema()

    words = readStream.select(explode(split(readStream.value,'  ')).alias('word'))
    wordCounts = words.groupBy('word').count().orderBy('count')

    query = wordCounts.writeStream.outputMode('complete').format('console').start().awaitTermination()

if __name__ == '__main__':
    main()

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题