下面是如何使用Spark functions ，可以使用spark转换给定的scala函数 functions 如下所示

import org.apache.spark.sql.functions._

df.withColumn("n_trip",
  array_max(
    transform(
      filter(
        split(array_join($"trip", " "), "co"),
        (col: Column) => (col =!= "" || col =!= null)
      ), (col: Column) => size(split(trim(col), " "))
    )
  ))
  .withColumn("n_trip", when($"n_trip".isNull, 0).otherwise($"n_trip"))
  .show(false)

输出：

+-----------+--------------------+------+
|passengerId|trip                |n_trip|
+-----------+--------------------+------+
|10096      |[co]                |0     |
|10351      |[pk]                |1     |
|10436      |[co, co, cn, tj, us]|3     |
|1090       |[dk, tj, jo, jo, ch]|5     |
|11078      |[pk, no, fr]        |3     |
|11332      |[sg, cn, co, bm]    |2     |
|11563      |[us, sg, th, cn]    |4     |
|1159       |[ca, cl, il, sg]    |4     |
|11722      |[dk, dk, pk, sg]    |4     |
|11888      |[au, se, ca, tj]    |4     |
|12394      |[dk, nl]            |2     |
|12529      |[no, be]            |2     |
|12847      |[cn]                |1     |
|13192      |[cn, tk, cg, uk]    |4     |
|13282      |[co, us, iq]        |2     |
|13442      |[cn, pk, jo, us, ch]|5     |
|13610      |[be, ar, tj, no, ch]|5     |
|13772      |[be, at]            |2     |
|13865      |[be, th, cn]        |3     |
|14157      |[sg]                |1     |
+-----------+--------------------+------+

您可以编写一个用户定义函数（udf）来计算所需的内容。有很多方法可以计算最长的序列。一个简单的方法是在 "co" ，计算每个子序列的大小并取最大值。

val longuest_seq = udf((x : Seq[String]) => {
    x.reduce(_ +" "+_)
     .split(" *co *")
     .map(_.count(_ == ' ') + 1)
     .max
})

val df = Seq(
    (1, Array("x", "y", "co", "z")),
    (2, Array("x")),
    (3, Array("co", "t")),
    (4, Array("a", "b", "c", "d", "co", "e"))
).toDF("id", "trip")

df.withColumn("n_trips", longuest_seq('trip)).show

这就产生了

+---+-------------------+-------+
| id|               trip|n_trips|
+---+-------------------+-------+
|  1|      [x, y, co, z]|      2|
|  2|                [x]|      1|
|  3|            [co, t]|      1|
|  4|[a, b, c, d, co, e]|      4|
+---+-------------------+-------+