在pig中进行单词计数

ghhaqwfi  于 2021-06-25  发布在  Pig
关注(0)|答案(2)|浏览(280)

我已经处理了以下表格中的数据:

( id ,{ bag of words})

例如:

(foobar, {(foo), (foo),(foobar),(bar)})
(foo,{(bar),(bar)})

以此类推。。你给了我:

processed: {id: chararray,tokens: {tuple_of_tokens: (token: chararray)}}

现在我想要的是。。同时计算一个单词出现在该数据中的次数,并将其输出为

foobar, foo, 2
foobar,foobar,1
foobar,bar,1
foo,bar,2

and so on...

我在Pig里怎么做?

mwyxok5s

mwyxok5s1#

虽然你可以在纯Pig身上做这件事,但用自定义项做这件事应该效率更高。大致如下:

@outputschema('wordcounts: {T:(word:chararray, count:int)}')
def generate_wordcount(BAG):
    d = {}
    for word in BAG:
        if word in d:
            d[word] += 1
        else:
            d[word] = 1
    return d.items()

您可以这样使用这个自定义项:

REGISTER 'myudfs.py' USING jython AS myudfs ;

-- A: (id, words: {T:(word:chararray)})

B = FOREACH A GENERATE id, FLATTEN(myudfs.generate_wordcount(words)) ;
y0u0uwnf

y0u0uwnf2#

试试这个:

$ cat input 
foobar  foo
foobar  foo
foobar  foobar
foobar  bar
foo bar
foo bar

--preparing
inputs = LOAD 'input' AS (first: chararray, second: chararray);
grouped = GROUP inputs BY first;
formatted = FOREACH grouped GENERATE group, inputs.second AS second;
--what you need
flattened = FOREACH formatted GENERATE group, FLATTEN(second);
result = FOREACH (GROUP flattened BY (group, second)) GENERATE FLATTEN(group), COUNT(flattened);
DUMP result;

输出:

(foo,bar,2)
(foobar,bar,1)
(foobar,foo,2)
(foobar,foobar,1)

相关问题