hadoop arraywritable给我一个classcastexception

7d7tgy0s  于 2021-06-03  发布在  Hadoop
关注(0)|答案(1)|浏览(252)

编辑:问题解决了-我犯了一个相当愚蠢的错误。
我有一个mapreduce管道,由map、reduce、map和reduce组成。我对第一个reduce使用sequencefileoutputformat,对第二个map使用sequencefileinputformat。我看了一下它的用法,似乎我用对了。我在这里输入的类型是intwritable和IntPairaryWritable(一个自定义arraywritable子类,使用mahout中的intpairwritable)。问题是,当读取第二个Map中的IntPairaryWritable时,当我尝试获取单个IntPairWritable时,会出现classcastexception。我不确定这是因为我使用arraywritable类时出错,还是因为我使用的sequencefile{input,output}格式有问题。我在这里和其他地方看了很多例子,我觉得这两个都做对了,但还是有一个错误。有什么帮助吗?
具体内容:
这是我的第一节课:

public static class WalkIdReducer extends MapReduceBase implements
        Reducer<IntWritable, IntPairWritable, IntWritable, IntPairArrayWritable> {

    @Override
    public void reduce(IntWritable walk_id, Iterator<IntPairWritable> values,
            OutputCollector<IntWritable, IntPairArrayWritable> output,
            Reporter reporter) throws IOException {
        ArrayList<IntPairWritable> value_array = new ArrayList<IntPairWritable>();
        while (values.hasNext()) {
            value_array.add(values.next());
        }
        output.collect(walk_id, IntPairArrayWritable.fromArrayList(value_array));
    }
}

第二个mapper类:

public static class NodePairMapper extends MapReduceBase implements
        Mapper<IntWritable, IntPairArrayWritable, IntPairWritable, Text> {

    @Override
    public void map(IntWritable key, IntPairArrayWritable value,
            OutputCollector<IntPairWritable, Text> output,
            Reporter reporter) throws IOException {
        // The following line gives a ClassCastException;
        // See IntPairArrayWritable.toArrayList(), below
        ArrayList<IntPairWritable> values = value.toArrayList();
        // other unimportant stuff
    }
}

第一个mapreduce的作业配置的相关部分:

conf.setReducerClass(WalkIdReducer.class);
    conf.setOutputKeyClass(IntWritable.class);
    conf.setOutputValueClass(IntPairArrayWritable.class);
    conf.setOutputFormat(SequenceFileOutputFormat.class);

对于第二个mapreduce:

conf.setInputFormat(SequenceFileInputFormat.class);
    conf.setMapperClass(NodePairMapper.class);

最后,我的arraywritable子类:

public static class IntPairArrayWritable extends ArrayWritable
{
    // These two methods are what people say is all you need for
    // creating an ArrayWritable subclass
    public IntPairArrayWritable() {
        super(IntPairArrayWritable.class);
    }

    public IntPairArrayWritable(IntPairWritable[] values) {
        super(IntPairArrayWritable.class, values);
    }

    // Some convenience methods, so I can use ArrayLists in
    // other parts of the code
    public static IntPairArrayWritable fromArrayList(
            ArrayList<IntPairWritable> array) {
        IntPairArrayWritable writable = new IntPairArrayWritable();
        IntPairWritable[] values = new IntPairWritable[array.size()];
        for (int i=0; i<array.size(); i++) {
            values[i] = array.get(i);
        }
        writable.set(values);
        return writable;
    }

    public ArrayList<IntPairWritable> toArrayList() {
        ArrayList<IntPairWritable> array = new ArrayList<IntPairWritable>();
        for (Writable pair : this.get()) {
            // This line is what kills it.  I get a ClassCastException here.
            IntPairWritable int_pair = (IntPairWritable) pair;
            array.add(int_pair);
        }
        return array;
    }
}

我得到的具体错误如下:

java.lang.ClassCastException: WalkAnalyzer$IntPairArrayWritable cannot be cast to org.apache.mahout.common.IntPairWritable
at WalkAnalyzer$IntPairArrayWritable.toArrayList(WalkAnalyzer.java:231)
at WalkAnalyzer$NodePairMapper.map(WalkAnalyzer.java:84)
at WalkAnalyzer$NodePairMapper.map(WalkAnalyzer.java:77)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)

我很困惑,为什么从arraywritable得到的get()方法是 WalkAnalyzer$IntPairArrayWritable -我希望get()返回 IntPairArrayWritable ,如api中所述。
编辑
我发现了问题。我是如何为intpairarraywriteable编写构造函数的。我打电话来了 super(IntPairArrayWritable.class); 当我应该打电话的时候 super(IntPairWritable.class); . 代码实际上应该是这样的:

public static class IntPairArrayWritable extends ArrayWritable
{
    // These two methods are what people say is all you need for
    // creating an ArrayWritable subclass
    public IntPairArrayWritable() {
        super(IntPairWritable.class);
    }

    public IntPairArrayWritable(IntPairWritable[] values) {
        super(IntPairWritable.class, values);
    }
}

我想对arraywritable子类使用一个不那么明显混淆的名称是个好主意,这样就更容易发现错误。

4szc88ey

4szc88ey1#

检查导入语句中的intpairwritable。看起来您在Map程序中选择了错误的包名称,因此正在强制转换到一个不同的类,即使它的名称也是intpairwritable。

相关问题