scala中折叠列的更漂亮的方法?

fivyi3re  于 2021-07-12  发布在  Spark
关注(0)|答案(0)|浏览(144)

我有一个sql,我正试图用Dataframe在sparkscala中编写它

SELECT country, 
       Substr(substring_col, 1, 3) AS Code, 
       CASE 
         WHEN Substr(substring_col, 1, 9) = '238208700' THEN 
         'columnName1' 
         WHEN Substr(substring_col, 1, 9) = '240018000' THEN 'columnName2' 
         WHEN Substr(substring_col, 1, 9) = '240017531' THEN 'columnName3' 
         WHEN Substr(substring_col, 1, 9) = '240017001' 
               OR Substr(substring_col, 1, 9) = '240017301' 
               OR Substr(substring_col, 1, 9) = '240017302' THEN 'columnName4' 
         WHEN Substr(substring_col, 1, 9) = '240017211' THEN 'columnName5' 
         WHEN Substr(substring_col, 1, 9) = '248010160' 
               OR Substr(substring_col, 1, 9) = '248010241' 
               OR Substr(substring_col, 1, 9) = '248010420' THEN 'columnName6' 
         ELSE 'not_defined' 
       END                AS custom_column_name 
FROM   t_filtered

下面是代码,我试图使它看起来漂亮,但无法达到预期的结果

val substring_col = substring(col("col_name"),1,9)
val mapper = Map("columnName1"  -> List("238208700"),
  "columnName2" -> List("240017301", "240017001")
)

myDf.withColumn("custom_column_name" ,mapper.foldLeft(lit(""))((accu, mapperMap) => {
  when(substring_col isin mapperMap._2, mapperMap._1)
}))

我得到下面的错误

Unsupported literal type class scala.collection.immutable.$colon$colon List(238208700)
java.lang.RuntimeException: Unsupported literal type class scala.collection.immutable.$colon$colon List(238208700)

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题