将Spark Dataframe 写入Delta Lake

ovfsdjhp  于 2022-10-07  发布在  Spark
关注(0)|答案(2)|浏览(145)

我试图使用文档提供的示例代码将Spark Dataframe 转换为Delta格式,但总是收到这个奇怪的错误。你能帮帮忙或当导游吗?

df_sdf.write.format("delta").save("/mnt/.../delta/")

错误如下:

org.apache.spark.SparkException: Job aborted.

--------------------------------------------------------------------------- Py4JJavaError Traceback (most recent call last) <command-3011941952225495> in <module> ----> 1 df_sdf.write.format("delta").save("/mnt/.../delta/") /databricks/spark/python/pyspark/sql/readwriter.py in save(self, path, format, mode, partitionBy,**options) 737 self._jwrite.save() 738 else: --> 739 self._jwrite.save(path) 740 741 @since(1.4)
/databricks/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in call(self, *args) 1255 answer = self.gateway_client.send_command(command) 1256 return_value = get_return_value( -> 1257 answer, self.gateway_client, self.target_id, self.name) 1258 1259 for temp_arg in temp_args:

/databricks/spark/python/pyspark/sql/utils.py in deco(a, *kw)
ruarlubt

ruarlubt1#

试试这个:

df_sdf.write.format("delta").save("/mnt/.../delta/sdf")
ia2d9nvy

ia2d9nvy2#

我也犯了同样的错误,问题是我使用的是Spark 3.0预览版。我不得不将Spark版本改为2.4,问题得到了解决。

相关问题