我试图提交一个Spark在我的本地申请,我得到下面的错误。
Exception in thread "main" org.apache.spark.SparkException: Cannot load main class from JAR file: at org.apache.spark.deploy.SparkSubmitArguments.error(SparkSubmitArguments.scala:657)
at org.apache.spark.deploy.SparkSubmitArguments.loadEnvironmentArguments(SparkSubmitArguments.scala:221)
at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:116)
at org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$1.<init>(SparkSubmit.scala:907)
at org.apache.spark.deploy.SparkSubmit$$anon$2.parseArguments(SparkSubmit.scala:907)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:81)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
下面是我用来提交spark申请的命令
spark提交单词\u count.py
我不知道我到底错过了什么,任何帮助都将不胜感激
from pyspark.sql import SparkSession
from pyspark.sql.functions import explode
from pyspark.sql.functions import split
def main():
sparkSession = SparkSession.builder.appName("Word Count").getOrCreate()
sparkSession.sparkContext.setLogLevel("ERROR")
readStream = sparkSession.readStream.format('text').load(path)
print("-------------------------------------------------")
print("Streaming source ready: ", readStream.isStreaming)
readStream.printSchema()
words = readStream.select(explode(split(readStream.value,' ')).alias('word'))
wordCounts = words.groupBy('word').count().orderBy('count')
query = wordCounts.writeStream.outputMode('complete').format('console').start().awaitTermination()
if __name__ == '__main__':
main()
暂无答案!
目前还没有任何答案,快来回答吧!