使用pyspark或python进行模糊匹配

dgsult0t  于 2021-05-27  发布在  Spark
关注(0)|答案(1)|浏览(551)

我尝试使用pyspark或python进行模糊匹配,其中有两个列表。
一。城市标准值列表

Clarksburg 
Fremont 
San Leandro 
Albuquerque 
Columbus 
San Jose 
Martinez 
New York
Alhambra
Unknown
Las Vegas
Dublin
Niagara Falls

二。拼写错误的城市名称列表

Clarksburg 
Closed 10/97
Fre,Nont
Fremong
San L:Eandro
Albuquerue
Clmbs
Sanjse
Martinz
New Yrk
Alambra
00011
L Vegas
Vegas
Ssan jose
Nw Yrk
Colmbus
Klarkburg
Alburque
Dublin
Niegara F

现在,我想将拼写错误的城市名称与标准值列表进行匹配,并创建另一个具有适当匹配的列表。我在找下面的输出

Clarksburg - Clarksburg
Closed 10/97 - Unknown
Fre,Nont - Fremont
Fremong - Fremont
San L:Eandro - San Leandro
Albuquerue - Albuquerque
Clmbs -Columbus
Sanjse - San Jose
Martinz - Martinez
New Yrk - New York
Alambra - Alhambra
00011 - Unknown
L Vegas - Las Vegas
Vegas - Las Vegas
Ssan jose - San Jose
Nw Yrk - New York
Colmbus - Columbus
Klarkburg - Clarksburg
Alburque - Albuquerque
Dublin - Dublin
Niegara F - Niagara Falls

任何帮助都能帮到我。提前谢谢。

erhoui1w

erhoui1w1#

使用 fuzzywuzzy ,然后改变 threshold 要满足您的要求:

from fuzzywuzzy import process

threshold = 40

matchlist = [x for x in """
Clarksburg
Fremont
San Leandro
Albuquerque
Columbus
San Jose
Martinez
New York
Alhambra
Unknown
Las Vegas
Dublin
Niagara Falls
""".splitlines() if x]

checklist = [x for x in """
Clarksburg
Closed 10/97
Fre,Nont
Fremong
San L:Eandro
Albuquerue
Clmbs
Sanjse
Martinz
New Yrk
Alambra
00011
L Vegas
Vegas
Ssan jose
Nw Yrk
Colmbus
Klarkburg
Alburque
Dublin
Niegara F
""".splitlines() if x]

for check in checklist:
    match = process.extractOne(check, matchlist)
    print(f"{check} - {match[0] if match[1] > threshold else 'Unknown'}")

这给了我:

Clarksburg - Clarksburg
Closed 10/97 - Unknown
Fre,Nont - Fremont
Fremong - Fremont
San L:Eandro - San Leandro
Albuquerue - Albuquerque
Clmbs - Columbus
Sanjse - San Jose
Martinz - Martinez
New Yrk - New York
Alambra - Alhambra
00011 - Unknown
L Vegas - Las Vegas
Vegas - Las Vegas
Ssan jose - San Jose
Nw Yrk - New York
Colmbus - Columbus
Klarkburg - Clarksburg
Alburque - Albuquerque
Dublin - Dublin
Niegara F - Niagara Falls

相关问题