PySpark列正在追加UDF的参数值

eagi6jfj  于 2022-10-07  发布在  Spark
关注(0)|答案(0)|浏览(109)

我已经写了一个小程序,它正在工作,但它是添加参数值到列中,我不需要。

输入:

预期:
一个大写字母上方的图像变为大写字母

获得:

代码:


# !/usr/bin/env python

import sys
import logging
from pyspark.sql.types import StringType
from pyspark.sql import SparkSession
from pyspark.sql.functions import udf, lit
from functools import partial

def main():
    process_function = {'TOLOWER' : process_lower_function }
    spark = SparkSession.builder.master("local[1]").appName('intelligent').getOrCreate()
    data = [(0, "cards"), (1, "upper")]
    deptColumns = ["index", "data"]
    df = spark.createDataFrame(data=data, schema = deptColumns)
    print (df.show())
    process_rule = "pyspark,testing,upper"
    lower_function = partial(process_lower_function, process_rule)
    udf_parser = udf(lower_function)
    df = df.withColumn("ndata",udf_parser("data"))
    #df.write.csv("udf.csv")
    print (df.show())

def process_lower_function(text, lower_rule):
    print (lower_rule)
    for rule in lower_rule.split(","):
        if rule in text:
            text = text.replace(rule, rule.upper())
            logging.info(f"LOWER :: {text} -- {rule}")
    print (text)
    return text

if __name__ == "__main__":
    sys.exit(main())

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题