在databricks runtime 6.4中找不到spark 2.4.5 from撸csv函数

pw136qt2  于 2021-05-27  发布在  Spark
关注(0)|答案(1)|浏览(363)

我在databricks runtime cluster 6.4的sparkversion 2.4.5的org.apache.spark.sql.functions包中得到了一个from\ csv函数错误。我看到这个函数被添加了很久,有谁能告诉我是导入了一个错误的包还是我做错了什么?

i5desfxk

i5desfxk1#

spark 3.0.0中引入了它,您可以看到回购:
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/functions.scala

/**
   * Parses a column containing a CSV string into a `StructType` with the specified schema.
   * Returns `null`, in the case of an unparseable string.
   *
   * @param e a string column containing CSV data.
   * @param schema the schema to use when parsing the CSV string
   * @param options options to control how the CSV is parsed. accepts the same options and the
   *                CSV data source.
   *
   * @group collection_funcs
   * @since 3.0.0
   */
  def from_csv(e: Column, schema: StructType, options: Map[String, String]): Column = withExpr {
    CsvToStructs(schema, options, e.expr)
  }

以及jira门票,包括:

https://issues.apache.org/jira/browse/SPARK-25393

相关问题