发出spark submit命令时出错

9avjhtql  于 2021-05-16  发布在  Spark
关注(0)|答案(0)|浏览(280)

我试图运行一个spark提交命令,它报告了一个语法错误,但我看不出问题是什么。以下是错误消息:

user@xxxxxxxx4 bin % spark-submit mnmcount.py data/mnm_dataset.csv
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/usr/local/Cellar/apache-spark/3.0.1/libexec/jars/spark-unsafe_2.12-3.0.1.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
20/12/23 10:41:29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

**File "/Users/xxxxxxxx/Downloads/spark-3.0.1-bin-hadoop2.7/bin/mnmcount.py", line 7

    print("Usage: mnmcount <file>", file=sys.stderr)
                                        ^
SyntaxError: invalid syntax**
log4j:WARN No appenders could be found for logger (org.apache.spark.util.ShutdownHookManager).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
user@xxxxxxxx4 bin %

下面是相关的python代码片段

import sys
from pyspark.sql import SparkSession
from pyspark.sql.functions import count

if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("Usage: mnmcount <file>", file=sys.stderr)
        sys.exit(-1)

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题