使用apache pig的pivot表

rqqzpn5f  于 2021-06-21  发布在  Pig
关注(0)|答案(2)|浏览(292)

我想知道是否有可能在apachepig中一次透视一个表。
输入:

Id    Column1 Column2 Column3
1      Row11    Row12   Row13
2      Row21    Row22   Row23

输出:

Id    Name     Value
1     Column1  Row11
1     Column2  Row12
1     Column3  Row13
2     Column1  Row21
2     Column2  Row22
2     Column3  Row23

真正的数据有几十列。
我可以用awk一次性完成,然后用hadoop流运行它。但我的大部分代码是apachepig,所以我想知道是否有可能在pig中高效地完成它。

5sxhfpxr

5sxhfpxr1#

你可以用两种方法:1。编写一个返回一包元组的udf。它将是最灵活的解决方案,但需要java代码;2像这样写一个僵硬的脚本:

inpt = load '/pig_fun/input/pivot.txt' as (Id, Column1, Column2, Column3);
bagged = foreach inpt generate Id, TOBAG(TOTUPLE('Column1', Column1), TOTUPLE('Column2', Column2), TOTUPLE('Column3', Column3)) as toPivot;
pivoted_1 = foreach bagged generate Id, FLATTEN(toPivot) as t_value;
pivoted = foreach pivoted_1 generate Id, FLATTEN(t_value);
dump pivoted;

运行此脚本得到以下结果:

(1,Column1,11)
(1,Column2,12)
(1,Column3,13)
(2,Column1,21)
(2,Column2,22)
(2,Column3,23)
(3,Column1,31)
(3,Column2,32)
(3,Column3,33)
ktca8awb

ktca8awb2#

我从id 1中删除了col3,以显示如何处理可选(null)数据
id名称值1 column1 row11 1 column2 row12 2 column1 row21 2 column2 row22 2 column3 row23
--Pig脚本.pig

data1       = load 'data.txt' using PigStorage() as (id:int, key:chararray, value:chararray);
grped       = group data1 by id;
pvt         = foreach grped {
    col1        = filter data1 by key =='Column1';
    col2        =filter data1  by key =='Column2';
    col3        =filter data1  by key =='Column3';
    generate flatten(group) as id,
        flatten(col1.value) as col1, 
        flatten(col2.value) as col2, 
        flatten((IsEmpty(col3.value) ? {('NULL')} : col3.value)) as col3; --HANDLE NULL
};
dump pvt;

结果:
(1,第11行,第12行,空)
(第2行,第21行,第22行,第23行)

相关问题