org.apache.hadoop.hive.ql.metadata.Hive.walkDirTree()方法的使用及代码示例

x33g5p2x  于2022-01-20 转载在 其他  
字(3.9k)|赞(0)|评价(0)|浏览(121)

本文整理了Java中org.apache.hadoop.hive.ql.metadata.Hive.walkDirTree()方法的一些代码示例,展示了Hive.walkDirTree()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。Hive.walkDirTree()方法的具体详情如下:
包路径:org.apache.hadoop.hive.ql.metadata.Hive
类名称:Hive
方法名:walkDirTree

Hive.walkDirTree介绍

[英]Walk through sub-directory tree to construct list bucketing location map.
[中]遍历子目录树以构建列表bucketing位置图。

代码示例

代码示例来源:origin: apache/hive

/**
 * Construct location map from path
 *
 * @param newPartPath
 * @param skewedInfo
 * @return
 * @throws IOException
 * @throws FileNotFoundException
 */
private Map<List<String>, String> constructListBucketingLocationMap(Path newPartPath,
  SkewedInfo skewedInfo) throws IOException, FileNotFoundException {
 Map<List<String>, String> skewedColValueLocationMaps = new HashMap<List<String>, String>();
 FileSystem fSys = newPartPath.getFileSystem(conf);
 walkDirTree(fSys.getFileStatus(newPartPath),
   fSys, skewedColValueLocationMaps, newPartPath, skewedInfo);
 return skewedColValueLocationMaps;
}

代码示例来源:origin: apache/drill

/**
 * Construct location map from path
 *
 * @param newPartPath
 * @param skewedInfo
 * @return
 * @throws IOException
 * @throws FileNotFoundException
 */
private Map<List<String>, String> constructListBucketingLocationMap(Path newPartPath,
  SkewedInfo skewedInfo) throws IOException, FileNotFoundException {
 Map<List<String>, String> skewedColValueLocationMaps = new HashMap<List<String>, String>();
 FileSystem fSys = newPartPath.getFileSystem(conf);
 walkDirTree(fSys.getFileStatus(newPartPath), fSys, skewedColValueLocationMaps, newPartPath,
   skewedInfo);
 return skewedColValueLocationMaps;
}

代码示例来源:origin: apache/hive

walkDirTree(child, fSys, skewedColValueLocationMaps, newPartPath, skewedInfo);

代码示例来源:origin: apache/drill

/**
 * Walk through sub-directory tree to construct list bucketing location map.
 *
 * @param fSta
 * @param fSys
 * @param skewedColValueLocationMaps
 * @param newPartPath
 * @param skewedInfo
 * @throws IOException
 */
private void walkDirTree(FileStatus fSta, FileSystem fSys,
  Map<List<String>, String> skewedColValueLocationMaps, Path newPartPath, SkewedInfo skewedInfo)
  throws IOException {
 /* Base Case. It's leaf. */
 if (!fSta.isDir()) {
  /* construct one location map if not exists. */
  constructOneLBLocationMap(fSta, skewedColValueLocationMaps, newPartPath, skewedInfo);
  return;
 }

 /* dfs. */
 FileStatus[] children = fSys.listStatus(fSta.getPath(), FileUtils.HIDDEN_FILES_PATH_FILTER);
 if (children != null) {
  for (FileStatus child : children) {
   walkDirTree(child, fSys, skewedColValueLocationMaps, newPartPath, skewedInfo);
  }
 }
}

代码示例来源:origin: com.facebook.presto.hive/hive-apache

/**
 * Construct location map from path
 *
 * @param newPartPath
 * @param skewedInfo
 * @return
 * @throws IOException
 * @throws FileNotFoundException
 */
private Map<List<String>, String> constructListBucketingLocationMap(Path newPartPath,
  SkewedInfo skewedInfo) throws IOException, FileNotFoundException {
 Map<List<String>, String> skewedColValueLocationMaps = new HashMap<List<String>, String>();
 FileSystem fSys = newPartPath.getFileSystem(conf);
 walkDirTree(fSys.getFileStatus(newPartPath), fSys, skewedColValueLocationMaps, newPartPath,
   skewedInfo);
 return skewedColValueLocationMaps;
}

代码示例来源:origin: com.facebook.presto.hive/hive-apache

/**
 * Walk through sub-directory tree to construct list bucketing location map.
 *
 * @param fSta
 * @param fSys
 * @param skewedColValueLocationMaps
 * @param newPartPath
 * @param skewedInfo
 * @throws IOException
 */
private void walkDirTree(FileStatus fSta, FileSystem fSys,
  Map<List<String>, String> skewedColValueLocationMaps, Path newPartPath, SkewedInfo skewedInfo)
  throws IOException {
 /* Base Case. It's leaf. */
 if (!fSta.isDir()) {
  /* construct one location map if not exists. */
  constructOneLBLocationMap(fSta, skewedColValueLocationMaps, newPartPath, skewedInfo);
  return;
 }

 /* dfs. */
 FileStatus[] children = fSys.listStatus(fSta.getPath(), FileUtils.HIDDEN_FILES_PATH_FILTER);
 if (children != null) {
  for (FileStatus child : children) {
   walkDirTree(child, fSys, skewedColValueLocationMaps, newPartPath, skewedInfo);
  }
 }
}

相关文章

微信公众号

最新文章

更多

Hive类方法