无法通过unittest中的pyspark连接到cassandra

qco9c6ql  于 2021-06-10  发布在  Cassandra
关注(0)|答案(0)|浏览(292)

我遇到的问题是pyspark无法加载 com.datastax.spark:spark-cassandra-connector_2.11:2.5.1 当单元测试(或pytesting)时。当在普通代码中使用包时,它会很好地加载库,但在unittest中,永远不会加载库,并且会发生错误。
错误:

java.lang.ClassNotFoundException: Failed to find data source: org.apache.spark.sql.cassandra. Please find packages at http://spark.apache.org/third-party-projects.html

代码段:

conf = SparkConf()
    conf.setAppName(XXX)
    conf.setMaster(XXX)
    conf.set("spark.jars.packages", "com.datastax.spark:spark-cassandra-connector_2.11:2.5.1")

    sc = SparkContext(conf=conf)
    spark = SparkSession(sc)

    df = spark.read.format("org.apache.spark.sql.cassandra") \
        .option("keyspace", "system") \
        .option("table", "local") \
        .option("spark.cassandra.auth.username", XXX) \
        .option("spark.cassandra.auth.password", XXX) \
        .option("spark.cassandra.connection.host", XXX) \
        .load()

有什么解决办法吗?Cassandra?

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题