parquet.schema.MessageType.getType()方法的使用及代码示例

x33g5p2x  于2022-01-25 转载在 其他  
字(9.2k)|赞(0)|评价(0)|浏览(138)

本文整理了Java中parquet.schema.MessageType.getType()方法的一些代码示例，展示了MessageType.getType()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。MessageType.getType()方法的具体详情如下：
包路径：parquet.schema.MessageType
类名称：MessageType
方法名：getType

MessageType.getType介绍

暂无

代码示例

代码示例来源：origin: prestodb/presto

public static parquet.schema.Type getParquetTypeByName(String columnName, MessageType messageType)
{
  if (messageType.containsField(columnName)) {
    return messageType.getType(columnName);
  }
  // parquet is case-sensitive, but hive is not. all hive columns get converted to lowercase
  // check for direct match above but if no match found, try case-insensitive match
  for (parquet.schema.Type type : messageType.getFields()) {
    if (type.getName().equalsIgnoreCase(columnName)) {
      return type;
    }
  }
  return null;
}

代码示例来源：origin: prestodb/presto

public static parquet.schema.Type getParquetType(HiveColumnHandle column, MessageType messageType, boolean useParquetColumnNames)
  {
    if (useParquetColumnNames) {
      return getParquetTypeByName(column.getName(), messageType);
    }

    if (column.getHiveColumnIndex() < messageType.getFieldCount()) {
      return messageType.getType(column.getHiveColumnIndex());
    }
    return null;
  }
}

代码示例来源：origin: prestodb/presto

.toArray(String[]::new);
ColumnPath columnPath = ColumnPath.get(path);
PrimitiveTypeName primitiveTypeName = messageType.getType(columnPath.toArray()).asPrimitiveType().getPrimitiveTypeName();
ColumnChunkMetaData column = ColumnChunkMetaData.get(
    columnPath,

代码示例来源：origin: julienledem/redelm

public Type getType(String ... path) {
 return getType(path, 0);
}

代码示例来源：origin: com.facebook.presto.hive/hive-apache

public Type getType(String ... path) {
 return getType(path, 0);
}

代码示例来源：origin: asakusafw/asakusafw

private static List<Mapping> computeMappingByPosition(
    DataModelDescriptor target, MessageType source) {
  if (LOG.isDebugEnabled()) {
    LOG.debug(MessageFormat.format(
        "Mapping columns by their position: model={0}", //$NON-NLS-1$
        target.getDataModelClass().getName()));
  }
  List<ColumnDescriptor> sources = source.getColumns();
  List<? extends PropertyDescriptor> targets = target.getPropertyDescriptors();
  List<Mapping> mappings = new ArrayList<>();
  int limit = Math.min(sources.size(), targets.size());
  for (int i = 0; i < limit; i++) {
    ColumnDescriptor s = sources.get(i);
    Type sType = source.getType(s.getPath());
    PropertyDescriptor t = targets.get(i);
    mappings.add(new Mapping(s, sType, t));
  }
  for (int i = limit, n = sources.size(); i < n; i++) {
    ColumnDescriptor s = sources.get(i);
    Type sType = source.getType(s.getPath());
    mappings.add(new Mapping(s, sType, null));
  }
  for (int i = limit, n = targets.size(); i < n; i++) {
    mappings.add(new Mapping(null, null, targets.get(i)));
  }
  return mappings;
}

代码示例来源：origin: uk.co.nichesolutions.presto/presto-hive

private static parquet.schema.Type getParquetTypeByName(String columnName, MessageType messageType)
  {
    if (messageType.containsField(columnName)) {
      return messageType.getType(columnName);
    }
    // parquet is case-sensitive, but hive is not. all hive columns get converted to lowercase
    // check for direct match above but if no match found, try case-insensitive match
    for (Type type : messageType.getFields()) {
      if (type.getName().equalsIgnoreCase(columnName)) {
        return type;
      }
    }

    return null;
  }
}

代码示例来源：origin: julienledem/redelm

public List<ColumnDescriptor> getColumns() {
 List<String[]> paths = this.getPaths(0);
 List<ColumnDescriptor> columns = new ArrayList<ColumnDescriptor>(paths.size());
 for (String[] path : paths) {
  // TODO: optimize this
  columns.add(new ColumnDescriptor(path, getType(path).asPrimitiveType().getPrimitiveTypeName(), getMaxRepetitionLevel(path), getMaxDefinitionLevel(path)));
 }
 return columns;
}

代码示例来源：origin: com.facebook.presto.hive/hive-apache

private SchemaCompatibilityValidator(MessageType schema) {
 for (ColumnDescriptor cd : schema.getColumns()) {
  ColumnPath columnPath = ColumnPath.get(cd.getPath());
  columnsAccordingToSchema.put(columnPath, cd);
  OriginalType ot = schema.getType(cd.getPath()).getOriginalType();
  if (ot != null) {
   originalTypes.put(columnPath, ot);
  }
 }
}

代码示例来源：origin: com.facebook.presto.hive/hive-apache

/**
 * Searchs column names by index on a given Parquet file schema, and returns its corresponded
 * Parquet schema types.
 *
 * @param schema Message schema where to search for column names.
 * @param colNames List of column names.
 * @param colIndexes List of column indexes.
 * @return A MessageType object of the column names found.
 */
private static MessageType getSchemaByIndex(MessageType schema, List<String> colNames, List<Integer> colIndexes) {
 List<Type> schemaTypes = new ArrayList<Type>();
 for (Integer i : colIndexes) {
  if (i < colNames.size()) {
   if (i < schema.getFieldCount()) {
    schemaTypes.add(schema.getType(i));
   } else {
    //prefixing with '_mask_' to ensure no conflict with named
    //columns in the file schema
    schemaTypes.add(Types.optional(PrimitiveTypeName.BINARY).named("_mask_" + colNames.get(i)));
   }
  }
 }
 return new MessageType(schema.getName(), schemaTypes);
}

代码示例来源：origin: asakusafw/asakusafw

private static List<Mapping> computeMappingByName(
    DataModelDescriptor target, MessageType source) {
  if (LOG.isDebugEnabled()) {
    LOG.debug(MessageFormat.format(
        "Mapping columns by their name: model={0}", //$NON-NLS-1$
        target.getDataModelClass().getName()));
  }
  Set<PropertyDescriptor> rest = new LinkedHashSet<>(target.getPropertyDescriptors());
  List<Mapping> mappings = new ArrayList<>();
  for (ColumnDescriptor s : source.getColumns()) {
    String name = s.getPath()[0];
    Type sType = source.getType(s.getPath());
    PropertyDescriptor t = target.findPropertyDescriptor(name);
    if (t != null) {
      mappings.add(new Mapping(s, sType, t));
      rest.remove(t);
    } else {
      mappings.add(new Mapping(s, sType, null));
    }
  }
  for (PropertyDescriptor t : rest) {
    mappings.add(new Mapping(null, null, t));
  }
  return mappings;
}

代码示例来源：origin: com.facebook.presto.hive/hive-apache

public List<ColumnDescriptor> getColumns() {
 List<String[]> paths = this.getPaths(0);
 List<ColumnDescriptor> columns = new ArrayList<ColumnDescriptor>(paths.size());
 for (String[] path : paths) {
  // TODO: optimize this
  PrimitiveType primitiveType = getType(path).asPrimitiveType();
  columns.add(new ColumnDescriptor(
          path,
          primitiveType.getPrimitiveTypeName(),
          primitiveType.getTypeLength(),
          getMaxRepetitionLevel(path),
          getMaxDefinitionLevel(path)));
 }
 return columns;
}

代码示例来源：origin: com.twitter/parquet-cascading

public SchemaIntersection(MessageType fileSchema, Fields requestedFields) {
 if(requestedFields == Fields.UNKNOWN)
  requestedFields = Fields.ALL;
 Fields newFields = Fields.NONE;
 List<Type> newSchemaFields = new ArrayList<Type>();
 int schemaSize = fileSchema.getFieldCount();
 for (int i = 0; i < schemaSize; i++) {
  Type type = fileSchema.getType(i);
  Fields name = new Fields(type.getName());
  if(requestedFields.contains(name)) {
   newFields = newFields.append(name);
   newSchemaFields.add(type);
  }
 }
 this.sourceFields = newFields;
 this.requestedSchema = new MessageType(fileSchema.getName(), newSchemaFields);
}

代码示例来源：origin: uk.co.nichesolutions.presto/presto-hive

public static parquet.schema.Type getParquetType(HiveColumnHandle column, MessageType messageType, boolean useParquetColumnNames)
{
  if (useParquetColumnNames) {
    return getParquetTypeByName(column.getName(), messageType);
  }
  if (column.getHiveColumnIndex() < messageType.getFieldCount()) {
    return messageType.getType(column.getHiveColumnIndex());
  }
  return null;
}

代码示例来源：origin: julienledem/redelm

public ColumnDescriptor getColumnDescription(String[] path) {
 int maxRep = getMaxRepetitionLevel(path);
 int maxDef = getMaxDefinitionLevel(path);
 PrimitiveTypeName type = getType(path).asPrimitiveType().getPrimitiveTypeName();
 return new ColumnDescriptor(path, type, maxRep, maxDef);
}

代码示例来源：origin: com.facebook.presto.hive/hive-apache

public ColumnDescriptor getColumnDescription(String[] path) {
 int maxRep = getMaxRepetitionLevel(path);
 int maxDef = getMaxDefinitionLevel(path);
 PrimitiveType type = getType(path).asPrimitiveType();
 return new ColumnDescriptor(path, type.getPrimitiveTypeName(),
               type.getTypeLength(), maxRep, maxDef);
}

代码示例来源：origin: uk.co.nichesolutions.presto/presto-hive

ColumnChunkMetaData column = ColumnChunkMetaData.get(
    columnPath,
    messageType.getType(columnPath.toArray()).asPrimitiveType().getPrimitiveTypeName(),
    CompressionCodecName.fromParquet(metaData.codec),
    readEncodings(metaData.encodings),
    readStats(metaData.statistics, messageType.getType(columnPath.toArray()).asPrimitiveType().getPrimitiveTypeName()),
    metaData.data_page_offset,
    metaData.dictionary_page_offset,

代码示例来源：origin: com.facebook.presto.hive/hive-apache

ColumnChunkMetaData column = ColumnChunkMetaData.get(
  path,
  messageType.getType(path.toArray()).asPrimitiveType().getPrimitiveTypeName(),
  CompressionCodecName.fromParquet(metaData.codec),
  fromFormatEncodings(metaData.encodings),
  fromParquetStatistics(metaData.statistics, messageType.getType(path.toArray()).asPrimitiveType().getPrimitiveTypeName()),
  metaData.data_page_offset,
  metaData.dictionary_page_offset,

代码示例来源：origin: julienledem/redelm

@Test
 public void testNestedTypes() {
  MessageType schema = MessageTypeParser.parseMessageType(Paper.schema.toString());
  Type type = schema.getType("Links", "Backward");
  assertEquals(PrimitiveTypeName.INT64,
    type.asPrimitiveType().getPrimitiveTypeName());
  assertEquals(0, schema.getMaxRepetitionLevel("DocId"));
  assertEquals(1, schema.getMaxRepetitionLevel("Name"));
  assertEquals(2, schema.getMaxRepetitionLevel("Name", "Language"));
  assertEquals(0, schema.getMaxDefinitionLevel("DocId"));
  assertEquals(1, schema.getMaxDefinitionLevel("Links"));
  assertEquals(2, schema.getMaxDefinitionLevel("Links", "Backward"));
 }
}

内容来源于网络，如有侵权，请联系作者删除！

MessageType

相关文章

    查看更多
    微信公众号

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com