您可以使用以下选项

df.drop(df.columns(0)) -- for dropping last column in scala

df.drop(df.columns[-1]) -- for dropping last column in pyspark

一种解决方案是简单地删除最后一列，如下所示：

part
  .select(part.columns.dropRight(1).map(col) : _*)
  .show(false)
+---+--------------+---+
|_c0|_c1           |_c2|
+---+--------------+---+
|1  |Consumer Goods|101|
|2  |Marketing     |102|
+---+--------------+---+

另一种解决方案是将文件作为文本文件读取，然后像这样自己拆分：

val text = spark.read.text("file_name")
// Note that the split functions in java/scala/spark ignores a separator that ends
// a string, but that one that starts one
val size = text.head.getAs[String]("value").split("\\|").size

text
  .withColumn("value", split('value, "\\|"))
  .select((0 until size).map(i => 'value getItem i as s"_c$i") : _*)
  .show(false)
+---+--------------+---+
|_c0|_c1           |_c2|
+---+--------------+---+
|1  |Consumer Goods|101|
|2  |Marketing     |102|
+---+--------------+---+