org.apache.parquet.schema.MessageType.getMaxDefinitionLevel()方法的使用及代码示例

x33g5p2x  于2022-01-25 转载在 其他  
字(11.2k)|赞(0)|评价(0)|浏览(105)

本文整理了Java中org.apache.parquet.schema.MessageType.getMaxDefinitionLevel()方法的一些代码示例，展示了MessageType.getMaxDefinitionLevel()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。MessageType.getMaxDefinitionLevel()方法的具体详情如下：
包路径：org.apache.parquet.schema.MessageType
类名称：MessageType
方法名：getMaxDefinitionLevel

MessageType.getMaxDefinitionLevel介绍

暂无

代码示例

代码示例来源：origin: org.apache.parquet/parquet-column

/**
 * @param path an array of strings representing the name path in this type
 * @return the max repetition level that might be needed to encode the
 * type at 'path'.
 */
public int getMaxDefinitionLevel(String ... path) {
 return getMaxDefinitionLevel(path, 0) - 1;
}

代码示例来源：origin: org.lasersonlab.apache.parquet/parquet-column

/**
 * @param path an array of strings representing the name path in this type
 * @return the max repetition level that might be needed to encode the
 * type at 'path'.
 */
public int getMaxDefinitionLevel(String ... path) {
 return getMaxDefinitionLevel(path, 0) - 1;
}

代码示例来源：origin: Netflix/iceberg

@Override
public ParquetValueWriter<?> struct(GroupType struct,
                  List<ParquetValueWriter<?>> fieldWriters) {
 List<Type> fields = struct.getFields();
 List<ParquetValueWriter<?>> writers = Lists.newArrayListWithExpectedSize(fieldWriters.size());
 for (int i = 0; i < fields.size(); i += 1) {
  Type fieldType = struct.getType(i);
  int fieldD = type.getMaxDefinitionLevel(path(fieldType.getName()));
  writers.add(option(fieldType, fieldD, fieldWriters.get(i)));
 }
 return new RecordWriter(writers);
}

代码示例来源：origin: Netflix/iceberg

@Override
public ParquetValueWriter<?> struct(GroupType struct,
                  List<ParquetValueWriter<?>> fieldWriters) {
 List<Type> fields = struct.getFields();
 List<ParquetValueWriter<?>> writers = Lists.newArrayListWithExpectedSize(fieldWriters.size());
 for (int i = 0; i < fields.size(); i += 1) {
  Type fieldType = struct.getType(i);
  int fieldD = type.getMaxDefinitionLevel(path(fieldType.getName()));
  writers.add(option(fieldType, fieldD, fieldWriters.get(i)));
 }
 return new RecordWriter(writers);
}

代码示例来源：origin: org.lasersonlab.apache.parquet/parquet-column

public List<ColumnDescriptor> getColumns() {
 List<String[]> paths = this.getPaths(0);
 List<ColumnDescriptor> columns = new ArrayList<ColumnDescriptor>(paths.size());
 for (String[] path : paths) {
  // TODO: optimize this
  PrimitiveType primitiveType = getType(path).asPrimitiveType();
  columns.add(new ColumnDescriptor(
          path,
          primitiveType,
          getMaxRepetitionLevel(path),
          getMaxDefinitionLevel(path)));
 }
 return columns;
}

代码示例来源：origin: org.apache.parquet/parquet-column

public List<ColumnDescriptor> getColumns() {
 List<String[]> paths = this.getPaths(0);
 List<ColumnDescriptor> columns = new ArrayList<ColumnDescriptor>(paths.size());
 for (String[] path : paths) {
  // TODO: optimize this
  PrimitiveType primitiveType = getType(path).asPrimitiveType();
  columns.add(new ColumnDescriptor(
          path,
          primitiveType,
          getMaxRepetitionLevel(path),
          getMaxDefinitionLevel(path)));
 }
 return columns;
}

代码示例来源：origin: Netflix/iceberg

@Override
public ParquetValueReader<?> list(Types.ListType expectedList, GroupType array,
                 ParquetValueReader<?> elementReader) {
 GroupType repeated = array.getFields().get(0).asGroupType();
 String[] repeatedPath = currentPath();
 int repeatedD = type.getMaxDefinitionLevel(repeatedPath)-1;
 int repeatedR = type.getMaxRepetitionLevel(repeatedPath)-1;
 Type elementType = repeated.getType(0);
 int elementD = type.getMaxDefinitionLevel(path(elementType.getName()))-1;
 return new ListReader<>(repeatedD, repeatedR, option(elementType, elementD, elementReader));
}

代码示例来源：origin: Netflix/iceberg

@Override
public ParquetValueWriter<?> list(GroupType array, ParquetValueWriter<?> elementWriter) {
 GroupType repeated = array.getFields().get(0).asGroupType();
 String[] repeatedPath = currentPath();
 int repeatedD = type.getMaxDefinitionLevel(repeatedPath);
 int repeatedR = type.getMaxRepetitionLevel(repeatedPath);
 org.apache.parquet.schema.Type elementType = repeated.getType(0);
 int elementD = type.getMaxDefinitionLevel(path(elementType.getName()));
 return collections(repeatedD, repeatedR, option(elementType, elementD, elementWriter));
}

代码示例来源：origin: Netflix/iceberg

@Override
public ParquetValueReader<?> list(Types.ListType expectedList, GroupType array, ParquetValueReader<?> elementReader) {
 GroupType repeated = array.getFields().get(0).asGroupType();
 String[] repeatedPath = currentPath();
 int repeatedD = type.getMaxDefinitionLevel(repeatedPath) - 1;
 int repeatedR = type.getMaxRepetitionLevel(repeatedPath) - 1;
 Type elementType = repeated.getType(0);
 int elementD = type.getMaxDefinitionLevel(path(elementType.getName())) - 1;
 return new ArrayReader<>(repeatedD, repeatedR, option(elementType, elementD, elementReader));
}

代码示例来源：origin: org.apache.parquet/parquet-column

public ColumnDescriptor getColumnDescription(String[] path) {
 int maxRep = getMaxRepetitionLevel(path);
 int maxDef = getMaxDefinitionLevel(path);
 PrimitiveType type = getType(path).asPrimitiveType();
 return new ColumnDescriptor(path, type, maxRep, maxDef);
}

代码示例来源：origin: Netflix/iceberg

@Override
public ParquetValueReader<?> list(Types.ListType expectedList, GroupType array,
                 ParquetValueReader<?> elementReader) {
 GroupType repeated = array.getFields().get(0).asGroupType();
 String[] repeatedPath = currentPath();
 int repeatedD = type.getMaxDefinitionLevel(repeatedPath)-1;
 int repeatedR = type.getMaxRepetitionLevel(repeatedPath)-1;
 Type elementType = repeated.getType(0);
 int elementD = type.getMaxDefinitionLevel(path(elementType.getName()))-1;
 return new ArrayReader<>(repeatedD, repeatedR, option(elementType, elementD, elementReader));
}

代码示例来源：origin: Netflix/iceberg

@Override
public ParquetValueReader<?> list(Types.ListType expectedList, GroupType array,
                 ParquetValueReader<?> elementReader) {
 GroupType repeated = array.getFields().get(0).asGroupType();
 String[] repeatedPath = currentPath();
 int repeatedD = type.getMaxDefinitionLevel(repeatedPath)-1;
 int repeatedR = type.getMaxRepetitionLevel(repeatedPath)-1;
 Type elementType = repeated.getType(0);
 int elementD = type.getMaxDefinitionLevel(path(elementType.getName()))-1;
 return new ListReader<>(repeatedD, repeatedR, option(elementType, elementD, elementReader));
}

代码示例来源：origin: Netflix/iceberg

@Override
public ParquetValueWriter<?> list(GroupType array, ParquetValueWriter<?> elementWriter) {
 GroupType repeated = array.getFields().get(0).asGroupType();
 String[] repeatedPath = currentPath();
 int repeatedD = type.getMaxDefinitionLevel(repeatedPath);
 int repeatedR = type.getMaxRepetitionLevel(repeatedPath);
 org.apache.parquet.schema.Type elementType = repeated.getType(0);
 int elementD = type.getMaxDefinitionLevel(path(elementType.getName()));
 return collections(repeatedD, repeatedR, option(elementType, elementD, elementWriter));
}

代码示例来源：origin: Netflix/iceberg

@Override
public ParquetValueReader<?> map(Types.MapType expectedMap, GroupType map, ParquetValueReader<?> keyReader, ParquetValueReader<?> valueReader) {
 GroupType repeatedKeyValue = map.getFields().get(0).asGroupType();
 String[] repeatedPath = currentPath();
 int repeatedD = type.getMaxDefinitionLevel(repeatedPath) - 1;
 int repeatedR = type.getMaxRepetitionLevel(repeatedPath) - 1;
 Type keyType = repeatedKeyValue.getType(0);
 int keyD = type.getMaxDefinitionLevel(path(keyType.getName())) - 1;
 Type valueType = repeatedKeyValue.getType(1);
 int valueD = type.getMaxDefinitionLevel(path(valueType.getName())) - 1;
 return new MapReader<>(repeatedD, repeatedR,
   option(keyType, keyD, keyReader), option(valueType, valueD, valueReader));
}

代码示例来源：origin: org.lasersonlab.apache.parquet/parquet-column

public ColumnDescriptor getColumnDescription(String[] path) {
 int maxRep = getMaxRepetitionLevel(path);
 int maxDef = getMaxDefinitionLevel(path);
 PrimitiveType type = getType(path).asPrimitiveType();
 return new ColumnDescriptor(path, type, maxRep, maxDef);
}

代码示例来源：origin: Netflix/iceberg

@Override
public ParquetValueReader<?> map(Types.MapType expectedMap, GroupType map,
                 ParquetValueReader<?> keyReader,
                 ParquetValueReader<?> valueReader) {
 GroupType repeatedKeyValue = map.getFields().get(0).asGroupType();
 String[] repeatedPath = currentPath();
 int repeatedD = type.getMaxDefinitionLevel(repeatedPath)-1;
 int repeatedR = type.getMaxRepetitionLevel(repeatedPath)-1;
 Type keyType = repeatedKeyValue.getType(0);
 int keyD = type.getMaxDefinitionLevel(path(keyType.getName()))-1;
 Type valueType = repeatedKeyValue.getType(1);
 int valueD = type.getMaxDefinitionLevel(path(valueType.getName()))-1;
 return new MapReader<>(repeatedD, repeatedR,
   option(keyType, keyD, keyReader), option(valueType, valueD, valueReader));
}

代码示例来源：origin: Netflix/iceberg

@Override
public ParquetValueReader<?> map(Types.MapType expectedMap, GroupType map,
                 ParquetValueReader<?> keyReader,
                 ParquetValueReader<?> valueReader) {
 GroupType repeatedKeyValue = map.getFields().get(0).asGroupType();
 String[] repeatedPath = currentPath();
 int repeatedD = type.getMaxDefinitionLevel(repeatedPath)-1;
 int repeatedR = type.getMaxRepetitionLevel(repeatedPath)-1;
 Type keyType = repeatedKeyValue.getType(0);
 int keyD = type.getMaxDefinitionLevel(path(keyType.getName()))-1;
 Type valueType = repeatedKeyValue.getType(1);
 int valueD = type.getMaxDefinitionLevel(path(valueType.getName()))-1;
 return new MapReader<>(repeatedD, repeatedR,
   option(keyType, keyD, keyReader), option(valueType, valueD, valueReader));
}

代码示例来源：origin: Netflix/iceberg

@Override
public ParquetValueReader<?> map(Types.MapType expectedMap, GroupType map,
                 ParquetValueReader<?> keyReader,
                 ParquetValueReader<?> valueReader) {
 GroupType repeatedKeyValue = map.getFields().get(0).asGroupType();
 String[] repeatedPath = currentPath();
 int repeatedD = type.getMaxDefinitionLevel(repeatedPath)-1;
 int repeatedR = type.getMaxRepetitionLevel(repeatedPath)-1;
 Type keyType = repeatedKeyValue.getType(0);
 int keyD = type.getMaxDefinitionLevel(path(keyType.getName()))-1;
 Type valueType = repeatedKeyValue.getType(1);
 int valueD = type.getMaxDefinitionLevel(path(valueType.getName()))-1;
 return new MapReader<>(repeatedD, repeatedR,
   option(keyType, keyD, keyReader), option(valueType, valueD, valueReader));
}

代码示例来源：origin: Netflix/iceberg

@Override
public ParquetValueWriter<?> map(GroupType map,
                 ParquetValueWriter<?> keyWriter,
                 ParquetValueWriter<?> valueWriter) {
 GroupType repeatedKeyValue = map.getFields().get(0).asGroupType();
 String[] repeatedPath = currentPath();
 int repeatedD = type.getMaxDefinitionLevel(repeatedPath);
 int repeatedR = type.getMaxRepetitionLevel(repeatedPath);
 org.apache.parquet.schema.Type keyType = repeatedKeyValue.getType(0);
 int keyD = type.getMaxDefinitionLevel(path(keyType.getName()));
 org.apache.parquet.schema.Type valueType = repeatedKeyValue.getType(1);
 int valueD = type.getMaxDefinitionLevel(path(valueType.getName()));
 return maps(repeatedD, repeatedR,
   option(keyType, keyD, keyWriter), option(valueType, valueD, valueWriter));
}

代码示例来源：origin: Netflix/iceberg

@Override
public ParquetValueWriter<?> map(GroupType map,
                 ParquetValueWriter<?> keyWriter,
                 ParquetValueWriter<?> valueWriter) {
 GroupType repeatedKeyValue = map.getFields().get(0).asGroupType();
 String[] repeatedPath = currentPath();
 int repeatedD = type.getMaxDefinitionLevel(repeatedPath);
 int repeatedR = type.getMaxRepetitionLevel(repeatedPath);
 org.apache.parquet.schema.Type keyType = repeatedKeyValue.getType(0);
 int keyD = type.getMaxDefinitionLevel(path(keyType.getName()));
 org.apache.parquet.schema.Type valueType = repeatedKeyValue.getType(1);
 int valueD = type.getMaxDefinitionLevel(path(valueType.getName()));
 return maps(repeatedD, repeatedR,
   option(keyType, keyD, keyWriter), option(valueType, valueD, valueWriter));
}

内容来源于网络，如有侵权，请联系作者删除！

MessageType

相关文章

    查看更多
    微信公众号

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com