mapreduce异常

tf7tbtn2  于 2021-06-02  发布在  Hadoop
关注(0)|答案(2)|浏览(231)

我为mapreduce文本排序编写了这样的代码:

public static class SortMapper extends Mapper<Object, Text, Text, Text> {
    private Text citizenship = new Text();

    @Override
    public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
        citizenship.set(value.toString().split(",")[11]);
        context.write(citizenship, value);
    }
}

public static class PrintReducer extends Reducer<Text, Text, NullWritable, Text> {

    @Override
    protected void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException {
        Iterator<Text> valIt = values.iterator();

        while (valIt.hasNext()) {
            Text value = valIt.next();
            context.write(NullWritable.get(), value);
        }
    }
}

public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    Job job = Job.getInstance(conf, "Football Sort");
    job.setJarByClass(FootballSort.class);
    job.setMapperClass(SortMapper.class);
    job.setCombinerClass(PrintReducer.class);
    job.setReducerClass(PrintReducer.class);
    job.setMapOutputKeyClass(Text.class);
    job.setMapOutputValueClass(Text.class);
    job.setOutputKeyClass(NullWritable.class);
    job.setOutputValueClass(Text.class);
    FileInputFormat.addInputPath(job, new Path(args[0]));
    FileOutputFormat.setOutputPath(job, new Path(args[1]));
    System.exit(job.waitForCompletion(true) ? 0 : 1);
}

但它总是很吸引人
第26、34行出现ioexception原因:org.apache.hadoop.io.nullwriteable类不是org.apache.hadoop.io.text类

eivnm1vs

eivnm1vs1#

@abhinay:在这种情况下不能使用组合器。组合器是操作是可交换和关联的小型还原器,组合器的签名应与还原器匹配。如果组合器签名为“”,则会出现错误,因为还原器的输入键和值为--text和intwritable,但组合器的输出键和值类为text,nullwritable–2015年12月28日5:51
//job.setcombinerclass(printreducer.class);或者删除这个字符串是解决问题的方法

bihw5rsg

bihw5rsg2#

你的 mapper outputformat 与代码不匹配,在main方法中设置输出文本

job.setMapOutputKeyClass(Text.class);
    job.setMapOutputValueClass(Text.class);

但在你的Map里 public static class PrintReducer extends Reducer<Text, Text, NullWritable, Text> 你的孩子让他们 NullWritable TEXT

相关问题