org.apache.gobblin.util.AvroUtils.getDirectorySchemaHelper()方法的使用及代码示例

x33g5p2x  于2022-01-16 转载在 其他  
字(2.5k)|赞(0)|评价(0)|浏览(100)

本文整理了Java中org.apache.gobblin.util.AvroUtils.getDirectorySchemaHelper()方法的一些代码示例,展示了AvroUtils.getDirectorySchemaHelper()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。AvroUtils.getDirectorySchemaHelper()方法的具体详情如下:
包路径:org.apache.gobblin.util.AvroUtils
类名称:AvroUtils
方法名:getDirectorySchemaHelper

AvroUtils.getDirectorySchemaHelper介绍

暂无

代码示例

代码示例来源:origin: apache/incubator-gobblin

/**
 * Get the latest avro schema for a directory
 * @param directory the input dir that contains avro files
 * @param fs the {@link FileSystem} for the given directory.
 * @param latest true to return latest schema, false to return oldest schema
 * @return the latest/oldest schema in the directory
 * @throws IOException
 */
public static Schema getDirectorySchema(Path directory, FileSystem fs, boolean latest) throws IOException {
 Schema schema = null;
 try (Closer closer = Closer.create()) {
  List<FileStatus> files = getDirectorySchemaHelper(directory, fs);
  if (files == null || files.size() == 0) {
   LOG.warn("There is no previous avro file in the directory: " + directory);
  } else {
   FileStatus file = latest ? files.get(0) : files.get(files.size() - 1);
   LOG.debug("Path to get the avro schema: " + file);
   FsInput fi = new FsInput(file.getPath(), fs.getConf());
   GenericDatumReader<GenericRecord> genReader = new GenericDatumReader<>();
   schema = closer.register(new DataFileReader<>(fi, genReader)).getSchema();
  }
 } catch (IOException ioe) {
  throw new IOException("Cannot get the schema for directory " + directory, ioe);
 }
 return schema;
}

代码示例来源:origin: org.apache.gobblin/gobblin-utility

/**
 * Get the latest avro schema for a directory
 * @param directory the input dir that contains avro files
 * @param fs the {@link FileSystem} for the given directory.
 * @param latest true to return latest schema, false to return oldest schema
 * @return the latest/oldest schema in the directory
 * @throws IOException
 */
public static Schema getDirectorySchema(Path directory, FileSystem fs, boolean latest) throws IOException {
 Schema schema = null;
 try (Closer closer = Closer.create()) {
  List<FileStatus> files = getDirectorySchemaHelper(directory, fs);
  if (files == null || files.size() == 0) {
   LOG.warn("There is no previous avro file in the directory: " + directory);
  } else {
   FileStatus file = latest ? files.get(0) : files.get(files.size() - 1);
   LOG.debug("Path to get the avro schema: " + file);
   FsInput fi = new FsInput(file.getPath(), fs.getConf());
   GenericDatumReader<GenericRecord> genReader = new GenericDatumReader<>();
   schema = closer.register(new DataFileReader<>(fi, genReader)).getSchema();
  }
 } catch (IOException ioe) {
  throw new IOException("Cannot get the schema for directory " + directory, ioe);
 }
 return schema;
}

相关文章