hadoop reduce>还原

cngwdvgl  于 2021-06-02  发布在  Hadoop
关注(0)|答案(0)|浏览(307)

我在hadoop上运行一个作业。例行公事。
然而,我的工作是停留在Map100%减少67%。map reduce任务状态为“reduce>reduce”。
我的reduce()方法:

public static class MyReducer extends Reducer<DoubleWritable, Text, DoubleWritable, Text> {

    public void reduce(DoubleWritable key, Iterable<Text> values, Context context) throws IOException, InterruptedException {
        Text res = new Text("");

        for(Text value : values){
            if(!value.equals("")){
                res.set(res.toString() + value.toString() + " ");
            }
        }
        context.write(key, res);
    }
}

map()方法:

public static class TokenizerMapper extends Mapper<Object, Text, DoubleWritable, Text>{

    public void map(Object key, Text value, Context context) throws IOException, InterruptedException{
         StringTokenizer input = new StringTokenizer(value.toString());

         input.nextToken();
         Text page = new Text(input.nextToken());

         DoubleWritable vue = new DoubleWritable(Double.parseDouble(input.nextToken()) );
         context.write(vue, page);
    }
}

减少任务日志:

2014-10-27 10:15:37,284 DEBUG [communication thread] org.apache.hadoop.yarn.util.ProcfsBasedProcessTree: [ 29830 29825 ]
2014-10-27 10:15:37,285 DEBUG [IPC Parameter Sending Thread #0] org.apache.hadoop.ipc.Client: IPC Client (592959754) connection to /xxx.xx.xx.xx:51302 from job_1414399407473_0001 sending #614
2014-10-27 10:15:37,287 DEBUG [IPC Client (592959754) connection to /xxx.xx.xx.xx:51302 from job_1414399407473_0001] org.apache.hadoop.ipc.Client: IPC Client (592959754) connection to /xxx.xx.xx.xx:51302 from job_1414399407473_0001 got value #614
2014-10-27 10:15:37,288 DEBUG [communication thread] org.apache.hadoop.ipc.RPC: Call: statusUpdate 3
2014-10-27 10:15:40,296 DEBUG [communication thread] org.apache.hadoop.yarn.util.ProcfsBasedProcessTree: [ 29830 29825 ]
2014-10-27 10:15:40,296 DEBUG [IPC Parameter Sending Thread #0] org.apache.hadoop.ipc.Client: IPC Client (592959754) connection to /xxx.xx.xx.xx:51302 from job_1414399407473_0001 sending #615
2014-10-27 10:15:40,298 DEBUG [IPC Client (592959754) connection to /xxx.xx.xx.xx:51302 from job_1414399407473_0001] org.apache.hadoop.ipc.Client: IPC Client (592959754) connection to /xxx.xx.xx.xx:51302 from job_1414399407473_0001 got value #615
2014-10-27 10:15:40,298 DEBUG [communication thread] org.apache.hadoop.ipc.RPC: Call: statusUpdate 2

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题