org.apache.hadoop.hive.ql.exec.Utilities.getInputPathsTez()方法的使用及代码示例

x33g5p2x  于2022-02-01 转载在 其他  
字(3.5k)|赞(0)|评价(0)|浏览(77)

本文整理了Java中org.apache.hadoop.hive.ql.exec.Utilities.getInputPathsTez()方法的一些代码示例,展示了Utilities.getInputPathsTez()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。Utilities.getInputPathsTez()方法的具体详情如下:
包路径:org.apache.hadoop.hive.ql.exec.Utilities
类名称:Utilities
方法名:getInputPathsTez

Utilities.getInputPathsTez介绍

[英]On Tez we're not creating dummy files when getting/setting input paths. We let Tez handle the situation. We're also setting the paths in the AM so we don't want to depend on scratch dir and context.
[中]在Tez上,在获取/设置输入路径时,我们不会创建虚拟文件。我们让特兹来处理这个问题。我们还在AM中设置路径,这样我们就不想依赖scratch目录和上下文。

代码示例

代码示例来源:origin: apache/hive

Path[] getInputPaths(JobConf job) throws IOException {
 Path[] dirs;
 if (HiveConf.getVar(job, HiveConf.ConfVars.HIVE_EXECUTION_ENGINE).equals("spark")) {
  dirs = mrwork.getPathToPartitionInfo().keySet().toArray(new Path[]{});
 } else {
  dirs = FileInputFormat.getInputPaths(job);
  if (dirs.length == 0) {
   // on tez we're avoiding to duplicate the file info in FileInputFormat.
   if (HiveConf.getVar(job, HiveConf.ConfVars.HIVE_EXECUTION_ENGINE).equals("tez")) {
    try {
     List<Path> paths = Utilities.getInputPathsTez(job, mrwork);
     dirs = paths.toArray(new Path[paths.size()]);
    } catch (Exception e) {
     throw new IOException("Could not create input files", e);
    }
   } else {
    throw new IOException("No input paths specified in job");
   }
  }
 }
 StringInternUtils.internUriStringsInPathArray(dirs);
 return dirs;
}

代码示例来源:origin: apache/drill

Path[] getInputPaths(JobConf job) throws IOException {
 Path[] dirs;
 if (HiveConf.getVar(job, HiveConf.ConfVars.HIVE_EXECUTION_ENGINE).equals("spark")) {
  dirs = mrwork.getPathToPartitionInfo().keySet().toArray(new Path[]{});
 } else {
  dirs = FileInputFormat.getInputPaths(job);
  if (dirs.length == 0) {
   // on tez we're avoiding to duplicate the file info in FileInputFormat.
   if (HiveConf.getVar(job, HiveConf.ConfVars.HIVE_EXECUTION_ENGINE).equals("tez")) {
    try {
     List<Path> paths = Utilities.getInputPathsTez(job, mrwork);
     dirs = paths.toArray(new Path[paths.size()]);
    } catch (Exception e) {
     throw new IOException("Could not create input files", e);
    }
   } else {
    throw new IOException("No input paths specified in job");
   }
  }
 }
 StringInternUtils.internUriStringsInPathArray(dirs);
 return dirs;
}

代码示例来源:origin: apache/hive

conf.get(HiveConf.ConfVars.HIVETEZINPUTFORMAT.varname).equals(HiveInputFormat.class.getName())) {
MapWork mapWork = Utilities.getMapWork(jobConf);
List<Path> paths = Utilities.getInputPathsTez(jobConf, mapWork);
FileSystem fs = paths.get(0).getFileSystem(jobConf);
FileStatus[] fileStatuses = fs.listStatus(paths.get(0));
counterName = Utilities.getVertexCounterName(HiveInputCounters.RAW_INPUT_SPLITS.name(), vertexName);
tezCounters.findCounter(groupName, counterName).increment(splits.length);
final List<Path> paths = Utilities.getInputPathsTez(jobConf, work);
counterName = Utilities.getVertexCounterName(HiveInputCounters.INPUT_DIRECTORIES.name(), vertexName);
tezCounters.findCounter(groupName, counterName).increment(paths.size());

代码示例来源:origin: com.facebook.presto.hive/hive-apache

Path[] getInputPaths(JobConf job) throws IOException {
 Path[] dirs = FileInputFormat.getInputPaths(job);
 if (dirs.length == 0) {
  // on tez we're avoiding to duplicate the file info in FileInputFormat.
  if (HiveConf.getVar(job, HiveConf.ConfVars.HIVE_EXECUTION_ENGINE).equals("tez")) {
   try {
    List<Path> paths = Utilities.getInputPathsTez(job, mrwork);
    dirs = paths.toArray(new Path[paths.size()]);
   } catch (Exception e) {
    throw new IOException("Could not create input files", e);
   }
  } else {
   throw new IOException("No input paths specified in job");
  }
 }
 return dirs;
}

相关文章

微信公众号

最新文章

更多

Utilities类方法