使用架构加载数据时出现apache pig classcast异常

pn9klfpd  于 2021-06-25  发布在  Pig
关注(0)|答案(1)|浏览(303)

我有一个简单的tab分隔文件,其中包含一个pig模式,我正在尝试加载并添加两列。当我使用pigstorage的“--schema”选项加载时,添加操作失败,出现classcastexception。当我加载'--noschema'时,加法工作正常。为什么在前一种情况下,Pig只会例外地失败?
下面是一个示例文件,其中只有一行输入,值以制表符分隔:

a       1       1

模式“.pig\u schema”看起来像:

{"fields":[{"name":"str","type":55,"description":"autogenerated from Pig Field Schema","schema":null},{"name":"score","type":15,"description":"autogenerated from Pig Field Schema","schema":null},{"name":"count","type":15,"description":"autogenerated from Pig Field Schema","schema":null}],"version":0,"sortKeys":[],"sortKeyOrders":[]}

以下是grunt shell中的语句列表:

a1 = load '/local/workplace/data' using PigStorage(); --load with schema
describe a1; -- a1: {str: chararray,score: long,count: long}
b1 = foreach a1 generate score + count;
dump b1; -- throws exception
a2 = load '/local/workplace/data' using PigStorage('\t', '--noschema') as (str:chararray, score:long, count: long);
b2 = foreach a2 generate score+count; -- no exception
dump b2; -- works fine

引发的异常是:

org.apache.pig.backend.executionengine.ExecException: ERROR 0: Exception while executing [Add (Name: Add[long] - scope-34 Operator Key: scope-34) chi
ldren: [[POProject (Name: Project[long][0] - scope-32 Operator Key: scope-32) children: null at []], [POProject (Name: Project[long][1] - scope-33 Op
erator Key: scope-33) children: null at []]] at []]: java.lang.ClassCastException: org.apache.pig.data.DataByteArray cannot be cast to java.lang.Numb
er
        at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:338)
        at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:378)
        at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNextTuple(POForEach.java:298)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:282)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:277)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:210)
Caused by: java.lang.ClassCastException: org.apache.pig.data.DataByteArray cannot be cast to java.lang.Number
        at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.Add.genericGetNext(Add.java:100)
        at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.Add.getNextLong(Add.java:123)
        at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:323)

清管器版本:0.12.1

6fe3ivhb

6fe3ivhb1#

默认情况下,如果u点提供模式,则所有内容都被视为字节数组。

相关问题