mapreduce—如何在hadoop map reduce程序的mapper代码中获取输入文件名

vh0rcniy  于 2021-05-29  发布在  Hadoop
关注(0)|答案(2)|浏览(383)

我编写了一个带有两个输入参数的驱动程序代码:args[0]和args[1]。
我想使用args[0]即输入文件路径,作为Map程序代码中的文件路径之一,以便缓冲区读取器可以读取文件,如下所示:

{
public void map(LongWritable key, Text value, Context context)
        throws java.io.IOException, InterruptedException {

      ArrayList<String> al=new ArrayList<String>();
      ArrayList<String> nl =new ArrayList<String>();
       String line = null;

string filepath=args[0];字符串路径=args[1];

try {

            FileReader fileReader = 
                new FileReader(filePath);

            BufferedReader bufferedReader = 
                new BufferedReader(fileReader);

            while((line = bufferedReader.readLine()) != null)
            {
                if(line.contains("result") && line.contains("Complete_IND"))
                {

                    String pattern1 = "venk";
                    String pattern2 = "si";

                    Pattern p = Pattern.compile(Pattern.quote(pattern1) + "(.*?)" + Pattern.quote(pattern2));
                    Matcher m = p.matcher(line);
                    while (m.find()) {
                     System.out.println(m.group(1));
                    String  num=m.group(1);
                      int fin=Integer.parseInt(m.group(1));
                      if(fin>0)
                      {
                      System.out.println("number greater than zero"+fin);                    
                      al.add(num);
                      for(String obj:al)
                          {System.out.println("final obj"+obj);}}
                    } 
                }    
            }   

            // Always close files.
            bufferedReader.close();  
           // for(String obj:al){
            //  System.out.println(obj);
            //}
        }

        catch(FileNotFoundException ex) {
            System.out.println(
                "Unable to open file '" + 
                filePath + "'");                
        }
        catch(IOException ex) {
            System.out.println(
                "Error reading file '" 
                + filePath + "'");                  
            // Or we could just do this: 
            // ex.printStackTrace();
        }

       System.out.println("secondloop");
        try {

            // FileReader reads text files in the default encoding.
            FileReader fileReader = 
                    new FileReader(path);

                // Always wrap FileReader in BufferedReader.
                BufferedReader bufferedReader = 
                    new BufferedReader(fileReader);
            System.out.println("djdne"+path);

            // Always wrap FileReader in BufferedReader.
            BufferedReader bufferReader = 
                new BufferedReader(fileReader );
            System.out.println("djdne"+path);
            while((line = bufferReader.readLine()) != null)
            {
                System.out.println("djdne"+line);
                for(String obj:al)  {   

                if(line.contains(obj))
                        {
                    System.out.println("77");
                    System.out.println("1 obj is"+obj);
                    String pattern1 = "|";
                    String pattern2 = "|";

                    Pattern pt = Pattern.compile(Pattern.quote(pattern1) + "(.*?)" + Pattern.quote(pattern2));
                    Matcher mac = pt.matcher(line);
                    System.out.println("22");
                    while (mac.find()) {
                        System.out.println("33");
                     System.out.println(mac.group(1));
                     nl.add(mac.group(1));
                        for(int i=0;i<nl.size();i++)
                        {

                            Text si= new Text(al.get(i));                               

                            Text vi=new Text(nl.get(i));
                            context.write(si, vi);

                        }

                    }               
                        }

                        }

                }
            bufferReader.close();  

        }
        catch(FileNotFoundException ex) {
            System.out.println(
                "Unable to open file '" + 
                path + "'");                
        }
        catch(IOException ex) {
            System.out.println(
                "Error reading file '" 
                + path + "'");                  
            // Or we could just do this: 
            // ex.printStackTrace();
        }
        }
}
5ssjco0h

5ssjco0h1#

在配置Map器类中,可以获取文件名。然后你可以用它来阅读。更多细节示例

public void configure(JobConf job) {
    inputFile = job.get("mapred.input.file");
}
flvlnr44

flvlnr442#

您可以从Map器上下文和当前输入分割中获得它。

((FileSplit) context.getInputSplit()).getPath();

相关问题