pig脚本在使用宏两次时失败

mwecs4sa  于 2021-06-25  发布在  Pig
关注(0)|答案(0)|浏览(227)

我有一个pig脚本,它使用宏两次,在同一个关系上,但参数不同;对于每次使用,我在不同的字段中过滤相同的关系。宏的形状如下:

DEFINE doubleGroupJoin (mainField, mainRelation) returns out {
    valid = FILTER $mainRelation BY $mainField != '';
    r1 = FOREACH (GROUP valid BY $mainField) GENERATE 
            field1_1, field1_2, ...;
    r2 = FOREACH (GROUP valid BY ($mainField, otherfield1, ...) GENERATE
            field2_1, field2_2, ...;
    $out = FOREACH (JOIN R1 BY field1_1, R2 BY field1_2) GENERATE
            final1, final2, ...; 
}

在脚本中,我有以下内容:

-- Output1
finalR1 = doubleGroupJoin('field1', initialData);
STORE finalR1 INTO '$output/R1';
-- Output2
finalR2 = doubleGroupJoin('field2', initialData);
STORE finalR2 INTO '$output/R2';

如果我注解掉output1或output2块,则作业可以正常工作,但如果我尝试同时使用这两个块,则会出现以下错误:

java.lang.ClassCastException: org.apache.pig.data.BinSedesTuple cannot be cast to java.lang.String
    at org.apache.pig.backend.hadoop.HDataType.getWritableComparableTypes(HDataType.java:106)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Map.collect(PigGenericMapReduce.java:111)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:284)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:277)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)

在这里使用pig 0.12.0。有什么关于为什么会这样的建议吗?

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题