org.apache.gobblin.util.AvroUtils.getSchemaFromDataFile()方法的使用及代码示例

x33g5p2x  于2022-01-16 转载在 其他  
字(2.2k)|赞(0)|评价(0)|浏览(92)

本文整理了Java中org.apache.gobblin.util.AvroUtils.getSchemaFromDataFile()方法的一些代码示例,展示了AvroUtils.getSchemaFromDataFile()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。AvroUtils.getSchemaFromDataFile()方法的具体详情如下:
包路径:org.apache.gobblin.util.AvroUtils
类名称:AvroUtils
方法名:getSchemaFromDataFile

AvroUtils.getSchemaFromDataFile介绍

[英]Get Avro schema from an Avro data file.
[中]从Avro数据文件获取Avro架构。

代码示例

代码示例来源:origin: apache/incubator-gobblin

private static Schema getSchema(CombineFileSplit split, TaskAttemptContext cx, Integer idx) throws IOException {
 Schema schema = AvroJob.getInputKeySchema(cx.getConfiguration());
 if (schema != null) {
  return schema;
 }
 Path path = split.getPath(idx);
 FileSystem fs = path.getFileSystem(cx.getConfiguration());
 return AvroUtils.getSchemaFromDataFile(path, fs);
}

代码示例来源:origin: apache/incubator-gobblin

public static Schema getNewestSchemaFromSource(Path sourceDir, FileSystem fs) throws IOException {
 FileStatus[] files = fs.listStatus(sourceDir);
 Arrays.sort(files, new LastModifiedDescComparator());
 for (FileStatus status : files) {
  if (status.isDirectory()) {
   Schema schema = getNewestSchemaFromSource(status.getPath(), fs);
   if (schema != null)
    return schema;
  } else if (FilenameUtils.isExtension(status.getPath().getName(), AVRO)) {
   return AvroUtils.getSchemaFromDataFile(status.getPath(), fs);
  }
 }
 return null;
}

代码示例来源:origin: org.apache.gobblin/gobblin-compaction

private static Schema getSchema(CombineFileSplit split, TaskAttemptContext cx, Integer idx) throws IOException {
 Schema schema = AvroJob.getInputKeySchema(cx.getConfiguration());
 if (schema != null) {
  return schema;
 }
 Path path = split.getPath(idx);
 FileSystem fs = path.getFileSystem(cx.getConfiguration());
 return AvroUtils.getSchemaFromDataFile(path, fs);
}

代码示例来源:origin: org.apache.gobblin/gobblin-compaction

public static Schema getNewestSchemaFromSource(Path sourceDir, FileSystem fs) throws IOException {
 FileStatus[] files = fs.listStatus(sourceDir);
 Arrays.sort(files, new LastModifiedDescComparator());
 for (FileStatus status : files) {
  if (status.isDirectory()) {
   Schema schema = getNewestSchemaFromSource(status.getPath(), fs);
   if (schema != null)
    return schema;
  } else if (FilenameUtils.isExtension(status.getPath().getName(), AVRO)) {
   return AvroUtils.getSchemaFromDataFile(status.getPath(), fs);
  }
 }
 return null;
}

相关文章