org.apache.hadoop.hive.ql.metadata.Partition.getBucketCount()方法的使用及代码示例

x33g5p2x  于2022-01-26 转载在 其他  
字(9.7k)|赞(0)|评价(0)|浏览(124)

本文整理了Java中org.apache.hadoop.hive.ql.metadata.Partition.getBucketCount()方法的一些代码示例，展示了Partition.getBucketCount()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。Partition.getBucketCount()方法的具体详情如下：
包路径：org.apache.hadoop.hive.ql.metadata.Partition
类名称：Partition
方法名：getBucketCount

Partition.getBucketCount介绍

[英]The number of buckets is a property of the partition. However - internally we are just storing it as a property of the table as a short term measure.
[中]存储桶的数量是分区的一个属性。然而，在内部，我们只是将其存储为表的属性，作为短期度量。

代码示例

代码示例来源：origin: apache/hive

/**
 * get all paths for this partition in a sorted manner
 */
@SuppressWarnings("nls")
public FileStatus[] getSortedPaths() {
 try {
  // Previously, this got the filesystem of the Table, which could be
  // different from the filesystem of the partition.
  FileSystem fs = getDataLocation().getFileSystem(SessionState.getSessionConf());
  String pathPattern = getDataLocation().toString();
  if (getBucketCount() > 0) {
   pathPattern = pathPattern + "/*";
  }
  LOG.info("Path pattern = " + pathPattern);
  FileStatus srcs[] = fs.globStatus(new Path(pathPattern), FileUtils.HIDDEN_FILES_PATH_FILTER);
  Arrays.sort(srcs);
  for (FileStatus src : srcs) {
   LOG.info("Got file: " + src.getPath());
  }
  if (srcs.length == 0) {
   return null;
  }
  return srcs;
 } catch (Exception e) {
  throw new RuntimeException("Cannot get path ", e);
 }
}

代码示例来源：origin: apache/drill

/**
 * get all paths for this partition in a sorted manner
 */
@SuppressWarnings("nls")
public FileStatus[] getSortedPaths() {
 try {
  // Previously, this got the filesystem of the Table, which could be
  // different from the filesystem of the partition.
  FileSystem fs = getDataLocation().getFileSystem(SessionState.getSessionConf());
  String pathPattern = getDataLocation().toString();
  if (getBucketCount() > 0) {
   pathPattern = pathPattern + "/*";
  }
  LOG.info("Path pattern = " + pathPattern);
  FileStatus srcs[] = fs.globStatus(new Path(pathPattern), FileUtils.HIDDEN_FILES_PATH_FILTER);
  Arrays.sort(srcs);
  for (FileStatus src : srcs) {
   LOG.info("Got file: " + src.getPath());
  }
  if (srcs.length == 0) {
   return null;
  }
  return srcs;
 } catch (Exception e) {
  throw new RuntimeException("Cannot get path ", e);
 }
}

代码示例来源：origin: apache/hive

getBucketFilePathsOfPartition(p.getDataLocation(), pGraphContext);
int bucketCount = p.getBucketCount();
   p.getBucketCount() + ", whereas the number of files is " + fileNames.size();
 throw new SemanticException(
   ErrorMsg.BUCKETED_TABLE_METADATA_INCORRECT.getMsg(msg));

代码示例来源：origin: apache/drill

getBucketFilePathsOfPartition(p.getDataLocation(), pGraphContext);
int bucketCount = p.getBucketCount();
   p.getBucketCount() + ", whereas the number of files is " + fileNames.size();
 throw new SemanticException(
   ErrorMsg.BUCKETED_TABLE_METADATA_INCORRECT.getMsg(msg));

代码示例来源：origin: apache/hive

private boolean checkPartition(Partition partition,
  List<Integer> bucketPositionsDest,
  List<Integer> sortPositionsDest,
  List<Integer> sortOrderDest,
  int numBucketsDest) {
 // The bucketing and sorting positions should exactly match
 int numBuckets = partition.getBucketCount();
 if (numBucketsDest != numBuckets) {
  return false;
 }
 List<Integer> partnBucketPositions =
   getBucketPositions(partition.getBucketCols(), partition.getTable().getCols());
 List<Integer> sortPositions =
   getSortPositions(partition.getSortCols(), partition.getTable().getCols());
 List<Integer> sortOrder =
   getSortOrder(partition.getSortCols(), partition.getTable().getCols());
 return bucketPositionsDest.equals(partnBucketPositions) &&
   sortPositionsDest.equals(sortPositions) &&
   sortOrderDest.equals(sortOrder);
}

代码示例来源：origin: apache/hive

int num = sampleDescr.getNumerator();
int den = sampleDescr.getDenominator();
int bucketCount = part.getBucketCount();
String fullScanMsg = "";

代码示例来源：origin: apache/drill

int num = sampleDescr.getNumerator();
int den = sampleDescr.getDenominator();
int bucketCount = part.getBucketCount();
String fullScanMsg = "";

代码示例来源：origin: apache/hive

return getPath();
} else {
 int bcount = getBucketCount();
 if (bcount == 0) {
  return getPath();

代码示例来源：origin: apache/drill

private boolean checkPartition(Partition partition,
  List<Integer> bucketPositionsDest,
  List<Integer> sortPositionsDest,
  List<Integer> sortOrderDest,
  int numBucketsDest) {
 // The bucketing and sorting positions should exactly match
 int numBuckets = partition.getBucketCount();
 if (numBucketsDest != numBuckets) {
  return false;
 }
 List<Integer> partnBucketPositions =
   getBucketPositions(partition.getBucketCols(), partition.getTable().getCols());
 List<Integer> sortPositions =
   getSortPositions(partition.getSortCols(), partition.getTable().getCols());
 List<Integer> sortOrder =
   getSortOrder(partition.getSortCols(), partition.getTable().getCols());
 return bucketPositionsDest.equals(partnBucketPositions) &&
   sortPositionsDest.equals(sortPositions) &&
   sortOrderDest.equals(sortOrder);
}

代码示例来源：origin: apache/drill

return getPath();
} else {
 int bcount = getBucketCount();
 if (bcount == 0) {
  return getPath();

代码示例来源：origin: apache/hive

if (prunedPartList != null) {
 for (Partition p : prunedPartList.getPartitions()) {
  if (numBuckets != p.getBucketCount()) {

代码示例来源：origin: apache/drill

ctxt.setPartitions(prunedPartList);
for (Partition p : prunedPartList.getPartitions()) {
 if (nbuckets != p.getBucketCount()) {

代码示例来源：origin: apache/drill

} else if (alterTbl.getOp() == AlterTableTypes.ALTERBUCKETNUM) {
 if (part != null) {
  if (part.getBucketCount() == alterTbl.getNumberBuckets()) {
   return 0;

代码示例来源：origin: apache/hive

} else if (alterTbl.getOp() == AlterTableTypes.ALTERBUCKETNUM) {
 if (part != null) {
  if (part.getBucketCount() == alterTbl.getNumberBuckets()) {
   return null;

代码示例来源：origin: org.apache.hadoop.hive/hive-exec

/**
 * mapping from bucket number to bucket path
 */
// TODO: add test case and clean it up
@SuppressWarnings("nls")
public Path getBucketPath(int bucketNum) {
 try {
  // Previously, this got the filesystem of the Table, which could be
  // different from the filesystem of the partition.
  FileSystem fs = FileSystem.get(getPartitionPath().toUri(), Hive.get()
    .getConf());
  String pathPattern = getPartitionPath().toString();
  if (getBucketCount() > 0) {
   pathPattern = pathPattern + "/*";
  }
  LOG.info("Path pattern = " + pathPattern);
  FileStatus srcs[] = fs.globStatus(new Path(pathPattern));
  Arrays.sort(srcs);
  for (FileStatus src : srcs) {
   LOG.info("Got file: " + src.getPath());
  }
  if (srcs.length == 0) {
   return null;
  }
  return srcs[bucketNum].getPath();
 } catch (Exception e) {
  throw new RuntimeException("Cannot get bucket path for bucket "
    + bucketNum, e);
 }
}

代码示例来源：origin: com.facebook.presto.hive/hive-apache

/**
 * get all paths for this partition in a sorted manner
 */
@SuppressWarnings("nls")
public FileStatus[] getSortedPaths() {
 try {
  // Previously, this got the filesystem of the Table, which could be
  // different from the filesystem of the partition.
  FileSystem fs = getDataLocation().getFileSystem(SessionState.getSessionConf());
  String pathPattern = getDataLocation().toString();
  if (getBucketCount() > 0) {
   pathPattern = pathPattern + "/*";
  }
  LOG.info("Path pattern = " + pathPattern);
  FileStatus srcs[] = fs.globStatus(new Path(pathPattern), FileUtils.HIDDEN_FILES_PATH_FILTER);
  Arrays.sort(srcs);
  for (FileStatus src : srcs) {
   LOG.info("Got file: " + src.getPath());
  }
  if (srcs.length == 0) {
   return null;
  }
  return srcs;
 } catch (Exception e) {
  throw new RuntimeException("Cannot get path ", e);
 }
}

代码示例来源：origin: apache/lens

private void assertSamePartitions(List<Partition> parts1, List<Partition> parts2) {
 assertEquals(parts1.size(), parts2.size());
 for (int i = 0; i < parts1.size(); i++) {
  assertEquals(parts1.get(i).toString(), parts2.get(i).toString(), "toString differs at element " + i);
  assertEquals(parts1.get(i).getPath(), parts2.get(i).getPath(), "path differs at element " + i);
  assertEquals(parts1.get(i).getLocation(), parts2.get(i).getLocation(), "location differs at element " + i);
  assertEquals(parts1.get(i).getBucketCount(), parts2.get(i).getBucketCount(),
   "bucket count differs at element " + i);
  assertEquals(parts1.get(i).getCompleteName(), parts2.get(i).getCompleteName(),
   "complete name differs at element " + i);
 }
}

代码示例来源：origin: com.facebook.presto.hive/hive-apache

public boolean checkBucketedTable(Table tbl, ParseContext pGraphContext,
  PrunedPartitionList prunedParts) throws SemanticException {
 if (tbl.isPartitioned()) {
  List<Partition> partitions = prunedParts.getNotDeniedPartns();
  // construct a mapping of (Partition->bucket file names) and (Partition -> bucket number)
  if (!partitions.isEmpty()) {
   for (Partition p : partitions) {
    List<String> fileNames =
      AbstractBucketJoinProc.getBucketFilePathsOfPartition(p.getDataLocation(),
        pGraphContext);
    // The number of files for the table should be same as number of
    // buckets.
    int bucketCount = p.getBucketCount();
    if (fileNames.size() != 0 && fileNames.size() != bucketCount) {
     return false;
    }
   }
  }
 } else {
  List<String> fileNames =
    AbstractBucketJoinProc.getBucketFilePathsOfPartition(tbl.getDataLocation(),
      pGraphContext);
  Integer num = new Integer(tbl.getNumBuckets());
  // The number of files for the table should be same as number of buckets.
  if (fileNames.size() != 0 && fileNames.size() != num) {
   return false;
  }
 }
 return true;
}

代码示例来源：origin: com.facebook.presto.hive/hive-apache

private boolean checkPartition(Partition partition,
  List<Integer> bucketPositionsDest,
  List<Integer> sortPositionsDest,
  List<Integer> sortOrderDest,
  int numBucketsDest) {
 // The bucketing and sorting positions should exactly match
 int numBuckets = partition.getBucketCount();
 if (numBucketsDest != numBuckets) {
  return false;
 }
 List<Integer> partnBucketPositions =
   getBucketPositions(partition.getBucketCols(), partition.getTable().getCols());
 ObjectPair<List<Integer>, List<Integer>> partnSortPositionsOrder =
   getSortPositionsOrder(partition.getSortCols(), partition.getTable().getCols());
 return bucketPositionsDest.equals(partnBucketPositions) &&
   sortPositionsDest.equals(partnSortPositionsOrder.getFirst()) &&
   sortOrderDest.equals(partnSortPositionsOrder.getSecond());
}

代码示例来源：origin: org.apache.hadoop.hive/hive-exec

return getPath();
} else {
 int bcount = getBucketCount();
 if (bcount == 0) {
  return getPath();

内容来源于网络，如有侵权，请联系作者删除！

Partition

相关文章

    查看更多
    微信公众号

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多

    Partition类方法

    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com