hadoop作业控制,第二个作业不起作用

tzxcd3kk  于 2021-05-27  发布在  Hadoop
关注(0)|答案(0)|浏览(187)

我试图工作2Map和减少工作,这是第二个工作是依赖于第一个。实际上,我正试图用第一个mapreduce作业来计算电影的收视率,然后用第二个来排序。但第二次Map和第二次裁员的某些原因并不奏效。
我的第一个Map绘制者

public class MovieRatingsMapper extends Mapper<LongWritable, Text, Text, IntWritable> {

    private static final IntWritable ONE = new IntWritable(1);

    @Override
    public void map(LongWritable longWritable, Text text, Context context) {
        String line = text.toString();
        String[] words = line.split("\t");

        context.write(new Text(words[1]), ONE); 
    }
}

第一减速器

public class MovieRatingsReducer extends Reducer<Text, IntWritable, Text, IntWritable> {

        @Override
        public void reduce(Text key, Iterable<IntWritable> values, Context context){
            //System.out.println("movie rating reducer is working");
            String word = key.toString();
            int totalCount = 0;
            for (IntWritable value : values) {
                int count = value.get();
                totalCount += count;
            }

            context.write(new Text(word), new IntWritable(totalCount));
        }
    }

第二个Map器

public class MovieRatingsSortMapper extends Mapper<Text, Text, IntWritable, Text> {

    IntWritable frequency = new IntWritable();

    @Override
    public void map(Text key, Text value, Context context) {

        System.out.println("Movie Rating sort Mapper is working");

        int newVal = Integer.parseInt(value.toString());
        frequency.set(newVal);

        context.write(frequency, key);
    }
}

二级减速器

public class MovieRatingsSortReducer extends Reducer<IntWritable, Text, IntWritable, Text> {

    Text word = new Text();

    @Override
    public void reduce(IntWritable key, Iterable<Text> values, Context context) {

        System.out.println("movie rating sort reducer is working");

        for(Text value : values) {
            word.set(value);
            context.write(key, word);
        }
    }
}

驱动程序,主要方法

public class MovireRatingsSortDriver extends Configured implements Tool {

    public int run(String[] args) throws Exception {

        Path inputDirPath = new Path("src/main/resources/input/movieratings/");
        Path outputDirPath = new Path("src/main/resources/output/movieratings/temp/");

        Path inputDirPath2 = new Path("src/main/resources/output/movieratings/temp/");
        Path outputDirPath2 = new Path("src/main/resources/output/movieratings/");

        Configuration conf = new Configuration();
        conf.set("fs.defaultFS", "file:/");
        conf.set("mapreduce.framework.name", "local");
        FileSystem fs = FileSystem.getLocal(conf);
        fs.delete(outputDirPath, true);
        fs.delete(outputDirPath2, true);
        fs.setWriteChecksum(false);

        JobControl jobControl = new JobControl("jobChain");
        Configuration conf1 = getConf();

        Job job1 = Job.getInstance(conf1);
        job1.setJarByClass(MovireRatingsSortDriver.class);
        job1.setJobName("MovieRatings");

        FileInputFormat.addInputPath(job1, inputDirPath);
        FileOutputFormat.setOutputPath(job1, outputDirPath);

        job1.setMapperClass(MovieRatingsMapper.class);
        job1.setCombinerClass(MovieRatingsReducer.class);
        job1.setNumReduceTasks(1);
        job1.setOutputKeyClass(Text.class);
        job1.setOutputValueClass(IntWritable.class);

        ControlledJob controlledJob1 = new ControlledJob(job1.getConfiguration());
        controlledJob1.setJob(job1);

        jobControl.addJob(controlledJob1);

        Configuration conf2 = getConf();

        Job job2 = Job.getInstance(conf2);
        job2.setJarByClass(MovireRatingsSortDriver.class);
        job2.setJobName("Sorter");

        FileInputFormat.addInputPath(job2, inputDirPath2);
        FileOutputFormat.setOutputPath(job2, outputDirPath2);

        job2.setMapperClass(MovieRatingsSortMapper.class);
        job2.setReducerClass(MovieRatingsSortReducer.class);

        job2.setOutputKeyClass(IntWritable.class);
        job2.setOutputValueClass(Text.class);
        job2.setInputFormatClass(KeyValueTextInputFormat.class);     

        job2.setNumReduceTasks(1);

        ControlledJob controlledJob2 = new ControlledJob(job2.getConfiguration());
        controlledJob2.setJob(job2);

        // make job2 dependent on job1
        controlledJob2.addDependingJob(controlledJob1);

        // add the job to the job control
        jobControl.addJob(controlledJob2);

        Thread jobControlThread = new Thread(jobControl);
        jobControlThread.start();

        while (!jobControl.allFinished()){
            Thread.sleep(500);
        }

        jobControl.stop();
        return 0;

    }

    public static void main(String[] args) throws Exception {
        int exitCode = ToolRunner.run(new MovireRatingsSortDriver(), args);
        System.exit(exitCode);

    }

}

第一个Map器和reducer运行良好,并将结果文件创建到/movieratings/temp,但第二个Map器从不在没有错误或任何信息的情况下启动。
你能告诉我为什么第二份工作不起作用吗。首先我想,在第一个reducer中我给出了结果,但在第二个mapper中我得到了输入,因为我改成了,但什么也没变。现在我没有主意了。

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题