org.apache.hadoop.hive.ql.exec.Utilities.createTmpDirs()方法的使用及代码示例

x33g5p2x  于2022-02-01 转载在 其他  
字(10.0k)|赞(0)|评价(0)|浏览(80)

本文整理了Java中org.apache.hadoop.hive.ql.exec.Utilities.createTmpDirs()方法的一些代码示例，展示了Utilities.createTmpDirs()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。Utilities.createTmpDirs()方法的具体详情如下：
包路径：org.apache.hadoop.hive.ql.exec.Utilities
类名称：Utilities
方法名：createTmpDirs

Utilities.createTmpDirs介绍

[英]Hive uses tmp directories to capture the output of each FileSinkOperator. This method creates all necessary tmp directories for FileSinks in the Mapwork.
[中]Hive使用tmp目录捕获每个FileSinkOperator的输出。此方法为Mapwork中的文件链接创建所有必要的tmp目录。

代码示例

代码示例来源：origin: apache/hive

/**
 * Hive uses tmp directories to capture the output of each FileSinkOperator.
 * This method creates all necessary tmp directories for FileSinks in the ReduceWork.
 *
 * @param conf Used to get the right FileSystem
 * @param rWork Used to find FileSinkOperators
 * @throws IOException
 */
@SuppressWarnings("unchecked")
public static void createTmpDirs(Configuration conf, ReduceWork rWork)
  throws IOException {
 if (rWork == null) {
  return;
 }
 List<Operator<? extends OperatorDesc>> ops
  = new LinkedList<Operator<? extends OperatorDesc>>();
 ops.add(rWork.getReducer());
 createTmpDirs(conf, ops);
}

代码示例来源：origin: apache/drill

/**
 * Hive uses tmp directories to capture the output of each FileSinkOperator.
 * This method creates all necessary tmp directories for FileSinks in the ReduceWork.
 *
 * @param conf Used to get the right FileSystem
 * @param rWork Used to find FileSinkOperators
 * @throws IOException
 */
@SuppressWarnings("unchecked")
public static void createTmpDirs(Configuration conf, ReduceWork rWork)
  throws IOException {
 if (rWork == null) {
  return;
 }
 List<Operator<? extends OperatorDesc>> ops
  = new LinkedList<Operator<? extends OperatorDesc>>();
 ops.add(rWork.getReducer());
 createTmpDirs(conf, ops);
}

代码示例来源：origin: apache/drill

/**
 * Hive uses tmp directories to capture the output of each FileSinkOperator.
 * This method creates all necessary tmp directories for FileSinks in the Mapwork.
 *
 * @param conf Used to get the right FileSystem
 * @param mWork Used to find FileSinkOperators
 * @throws IOException
 */
public static void createTmpDirs(Configuration conf, MapWork mWork)
  throws IOException {
 Map<Path, ArrayList<String>> pa = mWork.getPathToAliases();
 if (pa != null) {
  // common case: 1 table scan per map-work
  // rare case: smb joins
  HashSet<String> aliases = new HashSet<String>(1);
  List<Operator<? extends OperatorDesc>> ops =
    new ArrayList<Operator<? extends OperatorDesc>>();
  for (List<String> ls : pa.values()) {
   for (String a : ls) {
    aliases.add(a);
   }
  }
  for (String a : aliases) {
   ops.add(mWork.getAliasToWork().get(a));
  }
  createTmpDirs(conf, ops);
 }
}

代码示例来源：origin: apache/hive

/**
 * Hive uses tmp directories to capture the output of each FileSinkOperator.
 * This method creates all necessary tmp directories for FileSinks in the Mapwork.
 *
 * @param conf Used to get the right FileSystem
 * @param mWork Used to find FileSinkOperators
 * @throws IOException
 */
public static void createTmpDirs(Configuration conf, MapWork mWork)
  throws IOException {
 Map<Path, ArrayList<String>> pa = mWork.getPathToAliases();
 if (MapUtils.isNotEmpty(pa)) {
  // common case: 1 table scan per map-work
  // rare case: smb joins
  HashSet<String> aliases = new HashSet<String>(1);
  List<Operator<? extends OperatorDesc>> ops =
    new ArrayList<Operator<? extends OperatorDesc>>();
  for (List<String> ls : pa.values()) {
   for (String a : ls) {
    aliases.add(a);
   }
  }
  for (String a : aliases) {
   ops.add(mWork.getAliasToWork().get(a));
  }
  createTmpDirs(conf, ops);
 }
}

代码示例来源：origin: apache/drill

Utilities.setInputPaths(cloned, inputPaths);
Utilities.setMapWork(cloned, (MapWork) work, scratchDir, false);
Utilities.createTmpDirs(cloned, (MapWork) work);
if (work instanceof MergeFileWork) {
 MergeFileWork mergeFileWork = (MergeFileWork) work;
cloned.setBoolean("mapred.task.is.map", false);
Utilities.setReduceWork(cloned, (ReduceWork) work, scratchDir, false);
Utilities.createTmpDirs(cloned, (ReduceWork) work);
cloned.set(Utilities.MAPRED_REDUCER_CLASS, ExecReducer.class.getName());

代码示例来源：origin: apache/hive

Utilities.setInputPaths(cloned, inputPaths);
Utilities.setMapWork(cloned, mapWork, scratchDir, false);
Utilities.createTmpDirs(cloned, mapWork);
if (work instanceof MergeFileWork) {
 MergeFileWork mergeFileWork = (MergeFileWork) work;
cloned.setBoolean("mapred.task.is.map", false);
Utilities.setReduceWork(cloned, (ReduceWork) work, scratchDir, false);
Utilities.createTmpDirs(cloned, (ReduceWork) work);
cloned.set(Utilities.MAPRED_REDUCER_CLASS, ExecReducer.class.getName());

代码示例来源：origin: apache/drill

private Vertex createVertex(JobConf conf, ReduceWork reduceWork,
  LocalResource appJarLr, List<LocalResource> additionalLr, FileSystem fs,
  Path mrScratchDir, Context ctx) throws Exception {
 // set up operator plan
 conf.set(Utilities.INPUT_NAME, reduceWork.getName());
 Utilities.setReduceWork(conf, reduceWork, mrScratchDir, false);
 // create the directories FileSinkOperators need
 Utilities.createTmpDirs(conf, reduceWork);
 VertexExecutionContext vertexExecutionContext = createVertexExecutionContext(reduceWork);
 // create the vertex
 Vertex reducer = Vertex.create(reduceWork.getName(),
   ProcessorDescriptor.create(ReduceTezProcessor.class.getName()).
     setUserPayload(TezUtils.createUserPayloadFromConf(conf)),
   reduceWork.isAutoReduceParallelism() ?
     reduceWork.getMaxReduceTasks() :
     reduceWork.getNumReduceTasks(), getContainerResource(conf));
 reducer.setTaskEnvironment(getContainerEnvironment(conf, false));
 reducer.setExecutionContext(vertexExecutionContext);
 reducer.setTaskLaunchCmdOpts(getContainerJavaOpts(conf));
 Map<String, LocalResource> localResources = new HashMap<String, LocalResource>();
 localResources.put(getBaseName(appJarLr), appJarLr);
 for (LocalResource lr: additionalLr) {
  localResources.put(getBaseName(lr), lr);
 }
 reducer.addTaskLocalFiles(localResources);
 return reducer;
}

代码示例来源：origin: apache/hive

private Vertex createVertex(JobConf conf, ReduceWork reduceWork, FileSystem fs,
  Path mrScratchDir, Context ctx, Map<String, LocalResource> localResources)
    throws Exception {
 // set up operator plan
 conf.set(Utilities.INPUT_NAME, reduceWork.getName());
 Utilities.setReduceWork(conf, reduceWork, mrScratchDir, false);
 // create the directories FileSinkOperators need
 Utilities.createTmpDirs(conf, reduceWork);
 VertexExecutionContext vertexExecutionContext = createVertexExecutionContext(reduceWork);
 // create the vertex
 Vertex reducer = Vertex.create(reduceWork.getName(),
   ProcessorDescriptor.create(ReduceTezProcessor.class.getName()).
     setUserPayload(TezUtils.createUserPayloadFromConf(conf)),
   reduceWork.isAutoReduceParallelism() ?
     reduceWork.getMaxReduceTasks() :
     reduceWork.getNumReduceTasks(), getContainerResource(conf));
 reducer.setTaskEnvironment(getContainerEnvironment(conf, false));
 reducer.setExecutionContext(vertexExecutionContext);
 reducer.setTaskLaunchCmdOpts(getContainerJavaOpts(conf));
 reducer.addTaskLocalFiles(localResources);
 return reducer;
}

代码示例来源：origin: apache/drill

Utilities.createTmpDirs(conf, mapWork);

代码示例来源：origin: apache/hive

Utilities.createTmpDirs(conf, mapWork);

代码示例来源：origin: apache/hive

Utilities.createTmpDirs(job, mWork);
Utilities.createTmpDirs(job, rWork);

代码示例来源：origin: apache/drill

Utilities.createTmpDirs(job, mWork);
Utilities.createTmpDirs(job, rWork);

代码示例来源：origin: com.facebook.presto.hive/hive-apache

/**
 * Hive uses tmp directories to capture the output of each FileSinkOperator.
 * This method creates all necessary tmp directories for FileSinks in the ReduceWork.
 *
 * @param conf Used to get the right FileSystem
 * @param rWork Used to find FileSinkOperators
 * @throws IOException
 */
@SuppressWarnings("unchecked")
public static void createTmpDirs(Configuration conf, ReduceWork rWork)
  throws IOException {
 if (rWork == null) {
  return;
 }
 List<Operator<? extends OperatorDesc>> ops
  = new LinkedList<Operator<? extends OperatorDesc>>();
 ops.add(rWork.getReducer());
 createTmpDirs(conf, ops);
}

代码示例来源：origin: com.facebook.presto.hive/hive-apache

/**
 * Hive uses tmp directories to capture the output of each FileSinkOperator.
 * This method creates all necessary tmp directories for FileSinks in the Mapwork.
 *
 * @param conf Used to get the right FileSystem
 * @param mWork Used to find FileSinkOperators
 * @throws IOException
 */
public static void createTmpDirs(Configuration conf, MapWork mWork)
  throws IOException {
 Map<String, ArrayList<String>> pa = mWork.getPathToAliases();
 if (pa != null) {
  List<Operator<? extends OperatorDesc>> ops =
   new ArrayList<Operator<? extends OperatorDesc>>();
  for (List<String> ls : pa.values()) {
   for (String a : ls) {
    ops.add(mWork.getAliasToWork().get(a));
   }
  }
  createTmpDirs(conf, ops);
 }
}

代码示例来源：origin: com.facebook.presto.hive/hive-apache

Utilities.setInputPaths(cloned, inputPaths);
Utilities.setMapWork(cloned, (MapWork) work, scratchDir, false);
Utilities.createTmpDirs(cloned, (MapWork) work);
if (work instanceof MergeFileWork) {
 MergeFileWork mergeFileWork = (MergeFileWork) work;
cloned.setBoolean("mapred.task.is.map", false);
Utilities.setReduceWork(cloned, (ReduceWork) work, scratchDir, false);
Utilities.createTmpDirs(cloned, (ReduceWork) work);
cloned.set(Utilities.MAPRED_REDUCER_CLASS, ExecReducer.class.getName());

代码示例来源：origin: com.facebook.presto.hive/hive-apache

private Vertex createVertex(JobConf conf, ReduceWork reduceWork,
  LocalResource appJarLr, List<LocalResource> additionalLr, FileSystem fs,
  Path mrScratchDir, Context ctx) throws Exception {
 // set up operator plan
 conf.set(Utilities.INPUT_NAME, reduceWork.getName());
 Utilities.setReduceWork(conf, reduceWork, mrScratchDir, false);
 // create the directories FileSinkOperators need
 Utilities.createTmpDirs(conf, reduceWork);
 // create the vertex
 Vertex reducer = Vertex.create(reduceWork.getName(),
   ProcessorDescriptor.create(ReduceTezProcessor.class.getName()).
   setUserPayload(TezUtils.createUserPayloadFromConf(conf)),
     reduceWork.isAutoReduceParallelism() ? reduceWork.getMaxReduceTasks() : reduceWork
       .getNumReduceTasks(), getContainerResource(conf));
 reducer.setTaskEnvironment(getContainerEnvironment(conf, false));
 reducer.setTaskLaunchCmdOpts(getContainerJavaOpts(conf));
 Map<String, LocalResource> localResources = new HashMap<String, LocalResource>();
 localResources.put(getBaseName(appJarLr), appJarLr);
 for (LocalResource lr: additionalLr) {
  localResources.put(getBaseName(lr), lr);
 }
 reducer.addTaskLocalFiles(localResources);
 return reducer;
}

代码示例来源：origin: com.facebook.presto.hive/hive-apache

Utilities.createTmpDirs(conf, mapWork);

代码示例来源：origin: com.facebook.presto.hive/hive-apache

Utilities.createTmpDirs(job, mWork);
Utilities.createTmpDirs(job, rWork);

内容来源于网络，如有侵权，请联系作者删除！

utilities

相关文章

    查看更多
    微信公众号

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多

    Utilities类方法

    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com