您可以使用regex模式（源代码：java正则表达式来删除重复的单词）和 regexp_replace 要替换重复的单词：

val df = spark.sql("select 'The account account has been cancelled for the account account account and with the account' col")

df.show(false)
+-------------------------------------------------------------------------------------------+
|col                                                                                        |
+-------------------------------------------------------------------------------------------+
|The account account has been cancelled for the account account account and with the account|
+-------------------------------------------------------------------------------------------+

val df2 = df.withColumn("col", regexp_replace(col("col"), "\\b(\\w+)(\\b\\W+\\b\\1\\b)*", "$1"))

df2.show(false)
+-------------------------------------------------------------------+
|col                                                                |
+-------------------------------------------------------------------+
|The account has been cancelled for the account and with the account|
+-------------------------------------------------------------------+