org.apache.gobblin.util.AvroUtils.getFieldValue()方法的使用及代码示例

x33g5p2x  于2022-01-16 转载在 其他  
字(13.4k)|赞(0)|评价(0)|浏览(98)

本文整理了Java中org.apache.gobblin.util.AvroUtils.getFieldValue()方法的一些代码示例，展示了AvroUtils.getFieldValue()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。AvroUtils.getFieldValue()方法的具体详情如下：
包路径：org.apache.gobblin.util.AvroUtils
类名称：AvroUtils
方法名：getFieldValue

AvroUtils.getFieldValue介绍

[英]Given a GenericRecord, this method will return the field specified by the path parameter. The fieldLocation parameter is an ordered string specifying the location of the nested field to retrieve. For example, field1.nestedField1 takes the the value of the field "field1", and retrieves the field "nestedField1" from it.
[中]给定GenericRecord，此方法将返回path参数指定的字段。fieldLocation参数是一个有序字符串，指定要检索的嵌套字段的位置。例如，field1。nestedField1获取字段“field1”的值，并从中检索字段“nestedField1”。

代码示例

代码示例来源：origin: apache/incubator-gobblin

private Object getAsObject(String fieldName) {
 Optional<Object> obj = AvroUtils.getFieldValue(record, fieldName);
 return obj.isPresent() ? obj.get() : null;
}

代码示例来源：origin: apache/incubator-gobblin

/**
  * Retrieve the value of the partition column field specified by this.partitionColumns
  */
 private Optional<Object> getWriterPartitionColumnValue(GenericRecord record) {
  if (!this.partitionColumns.isPresent()) {
   return Optional.absent();
  }

  for (String partitionColumn : this.partitionColumns.get()) {
   Optional<Object> fieldValue = AvroUtils.getFieldValue(record, partitionColumn);
   if (fieldValue.isPresent()) {
    return fieldValue;
   }
  }
  return Optional.absent();
 }
}

代码示例来源：origin: apache/incubator-gobblin

/**
 * Get the schema of a field
 *
 * @param record the input record which has the schema id
 * @param schemaIdLocation a dot separated location string the schema id
 * @return a schema referenced by the schema id
 */
protected Schema getFieldSchema(GenericRecord record, String schemaIdLocation) throws Exception {
 Optional<Object> schemaIdValue = AvroUtils.getFieldValue(record, schemaIdLocation);
 if (!schemaIdValue.isPresent()) {
  throw new Exception("Schema id with key " + schemaIdLocation + " not found in the record");
 }
 String schemaKey = String.valueOf(schemaIdValue.get());
 return (Schema) registry.getSchemaByKey(schemaKey);
}

代码示例来源：origin: apache/incubator-gobblin

/**
 * Get field value byte array
 *
 * @param record the input record which has the field
 * @param fieldLocation a dot separated location string to the field
 * @return the byte array of field value
 */
protected byte[] getFieldAsBytes(GenericRecord record, String fieldLocation) throws Exception {
 Optional<Object> bytesValue = AvroUtils.getFieldValue(record, fieldLocation);
 if (!bytesValue.isPresent()) {
  throw new Exception("Bytes value with key " + fieldLocation + " not found in the record");
 }
 ByteBuffer bb = (ByteBuffer) bytesValue.get();
 if (bb.hasArray()) {
  return bb.array();
 } else {
  byte[] payloadBytes = new byte[bb.remaining()];
  bb.get(payloadBytes);
  return payloadBytes;
 }
}

代码示例来源：origin: apache/incubator-gobblin

@Override
 public Iterable<Object> convertRecord(Schema outputSchema, GenericRecord inputRecord, WorkUnitState workUnit)
   throws DataConversionException {
  Optional<Object> field = AvroUtils.getFieldValue(inputRecord, this.fieldLocation);

  return field.isPresent() ? new SingleRecordIterable<>(field.get()) : new EmptyIterable<>();
 }
}

代码示例来源：origin: apache/incubator-gobblin

/**
  * Retrieves the specified field from the inputRecord, and checks if it is equal to the expected value
  * {@link #fieldValue}. If it is then it returns a {@link org.apache.gobblin.converter.SingleRecordIterable} for the input record.
  * Otherwise it returns a {@link EmptyIterable}.
  * {@inheritDoc}
  * @see org.apache.gobblin.converter.AvroToAvroConverterBase#convertRecord(org.apache.avro.Schema, org.apache.avro.generic.GenericRecord, org.apache.gobblin.configuration.WorkUnitState)
  */
 @Override
 public Iterable<GenericRecord> convertRecord(Schema outputSchema, GenericRecord inputRecord, WorkUnitState workUnit)
   throws DataConversionException {
  Optional<Object> fieldValue = AvroUtils.getFieldValue(inputRecord, this.fieldName);
  if (fieldValue.isPresent() && fieldValue.get().toString().equals(this.fieldValue)) {
   return new SingleRecordIterable<>(inputRecord);
  }
  return new EmptyIterable<>();
 }
}

代码示例来源：origin: apache/incubator-gobblin

@Override
 public Iterable<GenericRecord> convertRecord(Schema outputSchema, GenericRecord inputRecord, WorkUnitState workUnit)
   throws DataConversionException {
  GenericRecord genericRecord = new GenericData.Record(outputSchema);

  BiMap<String, String> inversedViewOfFieldsRenameMap = this.fieldsRenameMap.inverse();
  for (Schema.Field field : outputSchema.getFields()) {
   String curFieldName = field.name();
   String originalFieldName =
     inversedViewOfFieldsRenameMap.containsKey(curFieldName) ? inversedViewOfFieldsRenameMap.get(curFieldName)
       : curFieldName;
   if (this.nonMapFields.contains(originalFieldName)) {
    genericRecord.put(curFieldName, inputRecord.get(originalFieldName));
   } else {
    genericRecord.put(curFieldName,
      AvroUtils.getFieldValue(inputRecord, Joiner.on('.').join(this.mapFieldName, originalFieldName)).or(""));
   }
  }
  return new SingleRecordIterable<>(genericRecord);
 }
}

代码示例来源：origin: apache/incubator-gobblin

Optional<Object> optional = AvroUtils.getFieldValue(inputRecord, entry.getValue());
if (!optional.isPresent()) {
 throw new DataConversionException("Unable to get field value with location: " + entry.getValue());

代码示例来源：origin: apache/incubator-gobblin

@Override
 public Iterable<ObjectStoreDeleteOperation> convertRecord(Class<?> outputSchema, GenericRecord inputRecord,
   WorkUnitState workUnit) throws DataConversionException {
  Optional<Object> fieldValue = AvroUtils.getFieldValue(inputRecord, this.objectIdField);
  byte[] objectId;
  if (fieldValue.isPresent()) {
   if (fieldValue.get() instanceof Utf8) {
    objectId = ((Utf8) fieldValue.get()).getBytes();
   } else if (fieldValue.get() instanceof String) {
    objectId = ((String) fieldValue.get()).getBytes(Charsets.UTF_8);
   } else if (fieldValue.get() instanceof Long) {
    objectId = Longs.toByteArray((Long) fieldValue.get());
   } else if (fieldValue.get() instanceof Integer) {
    objectId = Ints.toByteArray((Integer) fieldValue.get());
   } else {
    objectId = (byte[]) fieldValue.get();
   }
   return new SingleRecordIterable<ObjectStoreDeleteOperation>(ObjectStoreOperationBuilder.deleteBuilder()
     .withObjectId(objectId).build());
  } else {
   throw new DataConversionException(String.format("Object Id field %s not found in record %s", this.objectIdField,
     inputRecord));
  }
 }
}

代码示例来源：origin: apache/incubator-gobblin

/**
 * This is a test to validate support of maps in {@link org.apache.gobblin.util.AvroUtils#getFieldValue(GenericRecord, String)}
 * and {@link org.apache.gobblin.util.AvroUtils#getFieldSchema(Schema, String)}
 * @throws IOException
 */
@Test
public void testGetObjectFromMap()
  throws IOException {
 final String TEST_FIELD_LOCATION = "Map.stringKey.Field";
 String avroFilePath = this.AVRO_DIR + "avroDir/avroUtilsTestFile.avro";
 GenericRecord record = getRecordFromFile(avroFilePath).get(0);
 Assert.assertEquals(AvroUtils.getFieldValue(record, TEST_FIELD_LOCATION).get().toString(), "stringValue2");
 Assert.assertEquals(AvroUtils.getFieldSchema(record.getSchema(), TEST_FIELD_LOCATION).get().getType(),
   Schema.Type.STRING);
}

代码示例来源：origin: apache/incubator-gobblin

Assert.fail(e.getMessage());
Object expected = AvroUtils.getFieldValue(outputRecord, "address.street_number").get();
Assert.assertTrue(outputRecord.get("addressStreet_number") == expected);
 Assert.fail(e.getMessage());
expected = AvroUtils.getFieldValue(outputRecord, "address.city").get();
Assert.assertTrue(outputRecord.get("addressCity") == expected);

代码示例来源：origin: apache/incubator-gobblin

@Test
 public void testUnionWithNull() {
  Schema nestedRecord = SchemaBuilder.record("nested").fields().requiredDouble("double")
    .requiredString("string").endRecord();
  Schema union = SchemaBuilder.unionOf().nullType().and().type(nestedRecord).endUnion();
  Schema schema = SchemaBuilder.record("record").fields().name("union").type(union).noDefault().endRecord();

  Schema doubleSchema = AvroUtils.getFieldSchema(schema, "union.double").get();
  Assert.assertEquals(doubleSchema.getType(), Schema.Type.DOUBLE);

  GenericRecord nested = new GenericData.Record(nestedRecord);
  nested.put("double", 10);
  nested.put("string", "testString");
  GenericRecord record = new GenericData.Record(schema);
  record.put("union", nested);

  String stringValue = AvroUtils.getFieldValue(record, "union.string").get().toString();
  Assert.assertEquals(stringValue, "testString");
 }
}

代码示例来源：origin: org.apache.gobblin/gobblin-core

private Object getAsObject(String fieldName) {
 Optional<Object> obj = AvroUtils.getFieldValue(record, fieldName);
 return obj.isPresent() ? obj.get() : null;
}

代码示例来源：origin: org.apache.gobblin/gobblin-core

/**
  * Retrieve the value of the partition column field specified by this.partitionColumns
  */
 private Optional<Object> getWriterPartitionColumnValue(GenericRecord record) {
  if (!this.partitionColumns.isPresent()) {
   return Optional.absent();
  }

  for (String partitionColumn : this.partitionColumns.get()) {
   Optional<Object> fieldValue = AvroUtils.getFieldValue(record, partitionColumn);
   if (fieldValue.isPresent()) {
    return fieldValue;
   }
  }
  return Optional.absent();
 }
}

代码示例来源：origin: org.apache.gobblin/gobblin-kafka-common

/**
 * Get the schema of a field
 *
 * @param record the input record which has the schema id
 * @param schemaIdLocation a dot separated location string the schema id
 * @return a schema referenced by the schema id
 */
protected Schema getFieldSchema(GenericRecord record, String schemaIdLocation) throws Exception {
 Optional<Object> schemaIdValue = AvroUtils.getFieldValue(record, schemaIdLocation);
 if (!schemaIdValue.isPresent()) {
  throw new Exception("Schema id with key " + schemaIdLocation + " not found in the record");
 }
 String schemaKey = String.valueOf(schemaIdValue.get());
 return (Schema) registry.getSchemaByKey(schemaKey);
}

代码示例来源：origin: org.apache.gobblin/gobblin-kafka-common

/**
 * Get field value byte array
 *
 * @param record the input record which has the field
 * @param fieldLocation a dot separated location string to the field
 * @return the byte array of field value
 */
protected byte[] getFieldAsBytes(GenericRecord record, String fieldLocation) throws Exception {
 Optional<Object> bytesValue = AvroUtils.getFieldValue(record, fieldLocation);
 if (!bytesValue.isPresent()) {
  throw new Exception("Bytes value with key " + fieldLocation + " not found in the record");
 }
 ByteBuffer bb = (ByteBuffer) bytesValue.get();
 if (bb.hasArray()) {
  return bb.array();
 } else {
  byte[] payloadBytes = new byte[bb.remaining()];
  bb.get(payloadBytes);
  return payloadBytes;
 }
}

代码示例来源：origin: org.apache.gobblin/gobblin-core

@Override
 public Iterable<Object> convertRecord(Schema outputSchema, GenericRecord inputRecord, WorkUnitState workUnit)
   throws DataConversionException {
  Optional<Object> field = AvroUtils.getFieldValue(inputRecord, this.fieldLocation);

  return field.isPresent() ? new SingleRecordIterable<>(field.get()) : new EmptyIterable<>();
 }
}

代码示例来源：origin: org.apache.gobblin/gobblin-core

/**
  * Retrieves the specified field from the inputRecord, and checks if it is equal to the expected value
  * {@link #fieldValue}. If it is then it returns a {@link org.apache.gobblin.converter.SingleRecordIterable} for the input record.
  * Otherwise it returns a {@link EmptyIterable}.
  * {@inheritDoc}
  * @see org.apache.gobblin.converter.AvroToAvroConverterBase#convertRecord(org.apache.avro.Schema, org.apache.avro.generic.GenericRecord, org.apache.gobblin.configuration.WorkUnitState)
  */
 @Override
 public Iterable<GenericRecord> convertRecord(Schema outputSchema, GenericRecord inputRecord, WorkUnitState workUnit)
   throws DataConversionException {
  Optional<Object> fieldValue = AvroUtils.getFieldValue(inputRecord, this.fieldName);
  if (fieldValue.isPresent() && fieldValue.get().toString().equals(this.fieldValue)) {
   return new SingleRecordIterable<>(inputRecord);
  }
  return new EmptyIterable<>();
 }
}

代码示例来源：origin: org.apache.gobblin/gobblin-core-base

@Override
 public Iterable<GenericRecord> convertRecord(Schema outputSchema, GenericRecord inputRecord, WorkUnitState workUnit)
   throws DataConversionException {
  GenericRecord genericRecord = new GenericData.Record(outputSchema);

  BiMap<String, String> inversedViewOfFieldsRenameMap = this.fieldsRenameMap.inverse();
  for (Schema.Field field : outputSchema.getFields()) {
   String curFieldName = field.name();
   String originalFieldName =
     inversedViewOfFieldsRenameMap.containsKey(curFieldName) ? inversedViewOfFieldsRenameMap.get(curFieldName)
       : curFieldName;
   if (this.nonMapFields.contains(originalFieldName)) {
    genericRecord.put(curFieldName, inputRecord.get(originalFieldName));
   } else {
    genericRecord.put(curFieldName,
      AvroUtils.getFieldValue(inputRecord, Joiner.on('.').join(this.mapFieldName, originalFieldName)).or(""));
   }
  }
  return new SingleRecordIterable<>(genericRecord);
 }
}

代码示例来源：origin: org.apache.gobblin/gobblin-core

@Override
 public Iterable<ObjectStoreDeleteOperation> convertRecord(Class<?> outputSchema, GenericRecord inputRecord,
   WorkUnitState workUnit) throws DataConversionException {
  Optional<Object> fieldValue = AvroUtils.getFieldValue(inputRecord, this.objectIdField);
  byte[] objectId;
  if (fieldValue.isPresent()) {
   if (fieldValue.get() instanceof Utf8) {
    objectId = ((Utf8) fieldValue.get()).getBytes();
   } else if (fieldValue.get() instanceof String) {
    objectId = ((String) fieldValue.get()).getBytes(Charsets.UTF_8);
   } else if (fieldValue.get() instanceof Long) {
    objectId = Longs.toByteArray((Long) fieldValue.get());
   } else if (fieldValue.get() instanceof Integer) {
    objectId = Ints.toByteArray((Integer) fieldValue.get());
   } else {
    objectId = (byte[]) fieldValue.get();
   }
   return new SingleRecordIterable<ObjectStoreDeleteOperation>(ObjectStoreOperationBuilder.deleteBuilder()
     .withObjectId(objectId).build());
  } else {
   throw new DataConversionException(String.format("Object Id field %s not found in record %s", this.objectIdField,
     inputRecord));
  }
 }
}

内容来源于网络，如有侵权，请联系作者删除！

AvroUtils

相关文章

    查看更多
    微信公众号

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com