org.apache.parquet.schema.MessageType.getType()方法的使用及代码示例

x33g5p2x  于2022-01-25 转载在 其他  
字(7.6k)|赞(0)|评价(0)|浏览(128)

本文整理了Java中org.apache.parquet.schema.MessageType.getType()方法的一些代码示例，展示了MessageType.getType()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。MessageType.getType()方法的具体详情如下：
包路径：org.apache.parquet.schema.MessageType
类名称：MessageType
方法名：getType

MessageType.getType介绍

暂无

代码示例

代码示例来源：origin: apache/hive

/**
 * Searches column names by indexes on a given Parquet file schema, and returns its corresponded
 * Parquet schema types.
 *
 * @param schema Message schema where to search for column names.
 * @param colNames List of column names.
 * @param colIndexes List of column indexes.
 * @return A MessageType object of the column names found.
 */
public static MessageType getSchemaByIndex(MessageType schema, List<String> colNames, List<Integer> colIndexes) {
 List<Type> schemaTypes = new ArrayList<Type>();
 for (Integer i : colIndexes) {
  if (i < colNames.size()) {
   if (i < schema.getFieldCount()) {
    schemaTypes.add(schema.getType(i));
   } else {
    //prefixing with '_mask_' to ensure no conflict with named
    //columns in the file schema
    schemaTypes.add(
     Types.optional(PrimitiveTypeName.BINARY).named("_mask_" + colNames.get(i)));
   }
  }
 }
 return new MessageType(schema.getName(), schemaTypes);
}

代码示例来源：origin: apache/hive

if (i < colNames.size()) {
 if (i < schema.getFieldCount()) {
  Type t = schema.getType(i);
  String tn = t.getName().toLowerCase();
  if (!prunedCols.containsKey(tn)) {
   schemaTypes.add(schema.getType(i));
  } else {
   if (t.isPrimitive()) {

代码示例来源：origin: apache/hive

Type parquetType = schema.getType(leaf.getColumnName());
 return buildFilterPredicateFromPredicateLeaf(leaf, parquetType);
} else {

代码示例来源：origin: org.apache.spark/spark-sql_2.10

" File schema:\n" + fileSchema);
builder.addFields(fileSchema.getType(s));

代码示例来源：origin: org.apache.spark/spark-sql_2.11

" File schema:\n" + fileSchema);
builder.addFields(fileSchema.getType(s));

代码示例来源：origin: org.apache.spark/spark-sql

" File schema:\n" + fileSchema);
builder.addFields(fileSchema.getType(s));

代码示例来源：origin: org.lasersonlab.apache.parquet/parquet-column

public Type getType(String ... path) {
 return getType(path, 0);
}

代码示例来源：origin: org.apache.parquet/parquet-column

public Type getType(String ... path) {
 return getType(path, 0);
}

代码示例来源：origin: io.prestosql/presto-parquet

public static org.apache.parquet.schema.Type getParquetTypeByName(String columnName, MessageType messageType)
{
  if (messageType.containsField(columnName)) {
    return messageType.getType(columnName);
  }
  // parquet is case-sensitive, but hive is not. all hive columns get converted to lowercase
  // check for direct match above but if no match found, try case-insensitive match
  for (org.apache.parquet.schema.Type type : messageType.getFields()) {
    if (type.getName().equalsIgnoreCase(columnName)) {
      return type;
    }
  }
  return null;
}

代码示例来源：origin: prestosql/presto

public static org.apache.parquet.schema.Type getParquetTypeByName(String columnName, MessageType messageType)
{
  if (messageType.containsField(columnName)) {
    return messageType.getType(columnName);
  }
  // parquet is case-sensitive, but hive is not. all hive columns get converted to lowercase
  // check for direct match above but if no match found, try case-insensitive match
  for (org.apache.parquet.schema.Type type : messageType.getFields()) {
    if (type.getName().equalsIgnoreCase(columnName)) {
      return type;
    }
  }
  return null;
}

代码示例来源：origin: org.lasersonlab.apache.parquet/parquet-column

public List<ColumnDescriptor> getColumns() {
 List<String[]> paths = this.getPaths(0);
 List<ColumnDescriptor> columns = new ArrayList<ColumnDescriptor>(paths.size());
 for (String[] path : paths) {
  // TODO: optimize this
  PrimitiveType primitiveType = getType(path).asPrimitiveType();
  columns.add(new ColumnDescriptor(
          path,
          primitiveType,
          getMaxRepetitionLevel(path),
          getMaxDefinitionLevel(path)));
 }
 return columns;
}

代码示例来源：origin: org.apache.parquet/parquet-column

public List<ColumnDescriptor> getColumns() {
 List<String[]> paths = this.getPaths(0);
 List<ColumnDescriptor> columns = new ArrayList<ColumnDescriptor>(paths.size());
 for (String[] path : paths) {
  // TODO: optimize this
  PrimitiveType primitiveType = getType(path).asPrimitiveType();
  columns.add(new ColumnDescriptor(
          path,
          primitiveType,
          getMaxRepetitionLevel(path),
          getMaxDefinitionLevel(path)));
 }
 return columns;
}

代码示例来源：origin: dremio/dremio-oss

private static Type getType(String[] pathSegments, int depth, MessageType schema) {
 Type type = schema.getType(Arrays.copyOfRange(pathSegments, 0, depth + 1));
 if (depth + 1 == pathSegments.length) {
  return type;
 } else {
  Preconditions.checkState(!type.isPrimitive());
  return new GroupType(type.getRepetition(), type.getName(), type.getOriginalType(), getType(pathSegments, depth + 1, schema));
 }
}

代码示例来源：origin: org.lasersonlab.apache.parquet/parquet-column

public ColumnDescriptor getColumnDescription(String[] path) {
 int maxRep = getMaxRepetitionLevel(path);
 int maxDef = getMaxDefinitionLevel(path);
 PrimitiveType type = getType(path).asPrimitiveType();
 return new ColumnDescriptor(path, type, maxRep, maxDef);
}

代码示例来源：origin: org.apache.parquet/parquet-column

public ColumnDescriptor getColumnDescription(String[] path) {
 int maxRep = getMaxRepetitionLevel(path);
 int maxDef = getMaxDefinitionLevel(path);
 PrimitiveType type = getType(path).asPrimitiveType();
 return new ColumnDescriptor(path, type, maxRep, maxDef);
}

代码示例来源：origin: org.apache.drill.exec/drill-java-exec

private static Type getType(String[] pathSegments, int depth, MessageType schema) {
 Type type = schema.getType(Arrays.copyOfRange(pathSegments, 0, depth + 1));
 if (depth + 1 == pathSegments.length) {
  return type;
 } else {
  Preconditions.checkState(!type.isPrimitive());
  return new GroupType(type.getRepetition(), type.getName(), getType(pathSegments, depth + 1, schema));
 }
}

代码示例来源：origin: com.twitter/scalding-parquet-cascading

public SchemaIntersection(MessageType fileSchema, Fields requestedFields) {
 if(requestedFields == Fields.UNKNOWN)
  requestedFields = Fields.ALL;
 Fields newFields = Fields.NONE;
 List<Type> newSchemaFields = new ArrayList<Type>();
 int schemaSize = fileSchema.getFieldCount();
 for (int i = 0; i < schemaSize; i++) {
  Type type = fileSchema.getType(i);
  Fields name = new Fields(type.getName());
  if(requestedFields.contains(name)) {
   newFields = newFields.append(name);
   newSchemaFields.add(type);
  }
 }
 this.sourceFields = newFields;
 this.requestedSchema = new MessageType(fileSchema.getName(), newSchemaFields);
}

代码示例来源：origin: com.twitter/scalding-parquet

public SchemaIntersection(MessageType fileSchema, Fields requestedFields) {
 if(requestedFields == Fields.UNKNOWN)
  requestedFields = Fields.ALL;
 Fields newFields = Fields.NONE;
 List<Type> newSchemaFields = new ArrayList<Type>();
 int schemaSize = fileSchema.getFieldCount();
 for (int i = 0; i < schemaSize; i++) {
  Type type = fileSchema.getType(i);
  Fields name = new Fields(type.getName());
  if(requestedFields.contains(name)) {
   newFields = newFields.append(name);
   newSchemaFields.add(type);
  }
 }
 this.sourceFields = newFields;
 this.requestedSchema = new MessageType(fileSchema.getName(), newSchemaFields);
}

代码示例来源：origin: org.apache.parquet/parquet-cascading3

public SchemaIntersection(MessageType fileSchema, Fields requestedFields) {
 if(requestedFields == Fields.UNKNOWN)
  requestedFields = Fields.ALL;
 Fields newFields = Fields.NONE;
 List<Type> newSchemaFields = new ArrayList<Type>();
 int schemaSize = fileSchema.getFieldCount();
 for (int i = 0; i < schemaSize; i++) {
  Type type = fileSchema.getType(i);
  Fields name = new Fields(type.getName());
  if(requestedFields.contains(name)) {
   newFields = newFields.append(name);
   newSchemaFields.add(type);
  }
 }
 this.sourceFields = newFields;
 this.requestedSchema = new MessageType(fileSchema.getName(), newSchemaFields);
}

代码示例来源：origin: prestosql/presto

public static org.apache.parquet.schema.Type getParquetType(HiveColumnHandle column, MessageType messageType, boolean useParquetColumnNames)
  {
    if (useParquetColumnNames) {
      return getParquetTypeByName(column.getName(), messageType);
    }

    if (column.getHiveColumnIndex() < messageType.getFieldCount()) {
      return messageType.getType(column.getHiveColumnIndex());
    }
    return null;
  }
}

内容来源于网络，如有侵权，请联系作者删除！

MessageType

相关文章

    查看更多
    微信公众号

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com