org.apache.gobblin.util.AvroUtils.serializeAsPath()方法的使用及代码示例

x33g5p2x  于2022-01-16 转载在 其他  
字(5.0k)|赞(0)|评价(0)|浏览(105)

本文整理了Java中org.apache.gobblin.util.AvroUtils.serializeAsPath()方法的一些代码示例,展示了AvroUtils.serializeAsPath()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。AvroUtils.serializeAsPath()方法的具体详情如下:
包路径:org.apache.gobblin.util.AvroUtils
类名称:AvroUtils
方法名:serializeAsPath

AvroUtils.serializeAsPath介绍

[英]Serialize a generic record as a relative Path. Useful for converting GenericRecord type keys into file system locations. For example {field1=v1, field2=v2} returns field1=v1/field2=v2 if includeFieldNames is true, or v1/v2 if it is false. Illegal HDFS tokens such as ':' and '\' will be replaced with ''. Additionally, parameter replacePathSeparators controls whether to replace path separators ('/') with ''.
[中]将泛型记录序列化为相对路径。用于将GenericRecord类型密钥转换为文件系统位置。例如{field1=v1,field2=v2}如果includeFieldNames为true,则返回field1=v1/field2=v2;如果includeFieldNames为false,则返回v1/v2。非法的HDFS令牌,如“:”和“\”将被替换为“\”。此外,参数replacePathSeparators控制是否将路径分隔符(“/”)替换为“\u1”。

代码示例

代码示例来源:origin: apache/incubator-gobblin

protected String getPartitionPath(State properties) {
 if (this.partition.isPresent()) {
  boolean includePartitionerFieldNames = properties.getPropAsBoolean(ForkOperatorUtils
    .getPropertyNameForBranch(WRITER_INCLUDE_PARTITION_IN_FILE_NAMES, this.branches, this.branch), false);
  boolean removePathSeparators = properties.getPropAsBoolean(ForkOperatorUtils
    .getPropertyNameForBranch(WRITER_REPLACE_PATH_SEPARATORS_IN_PARTITIONS, this.branches, this.branch), false);
  return AvroUtils.serializeAsPath(this.partition.get(), includePartitionerFieldNames, removePathSeparators).toString();
 } else {
  return null;
 }
}

代码示例来源:origin: apache/incubator-gobblin

@Test public void testSerializeAsPath() throws Exception {
 Schema schema =
   new Schema.Parser().parse("{\"type\":\"record\", \"name\":\"test\", " + "\"fields\":["
     + "{\"name\": \"name\", \"type\": \"string\"}, " + "{\"name\": \"title\", \"type\": \"string\"}" + "]}");
 GenericRecord partition = new GenericData.Record(schema);
 partition.put("name", "a/b:c\\d e");
 partition.put("title", "title");
 Assert.assertEquals(AvroUtils.serializeAsPath(partition, true, true), new Path("name=a_b_c_d_e/title=title"));
 Assert.assertEquals(AvroUtils.serializeAsPath(partition, false, true), new Path("a_b_c_d_e/title"));
 Assert.assertEquals(AvroUtils.serializeAsPath(partition, false, false), new Path("a/b_c_d_e/title"));
}

代码示例来源:origin: apache/incubator-gobblin

@Override
public State getFinalState() {
 State state = new State();
 try {
  for (Map.Entry<GenericRecord, DataWriter<D>> entry : this.partitionWriters.asMap().entrySet()) {
   if (entry.getValue() instanceof FinalState) {
    State partitionFinalState = ((FinalState) entry.getValue()).getFinalState();
    if (this.shouldPartition) {
     for (String key : partitionFinalState.getPropertyNames()) {
      // Prevent overwriting final state across writers
      partitionFinalState.setProp(key + "_" + AvroUtils.serializeAsPath(entry.getKey(), false, true),
        partitionFinalState.getProp(key));
     }
    }
    state.addAll(partitionFinalState);
   }
  }
  state.setProp("RecordsWritten", recordsWritten());
  state.setProp("BytesWritten", bytesWritten());
 } catch (Exception exception) {
  log.warn("Failed to get final state." + exception.getMessage());
  // If Writer fails to return bytesWritten, it might not be implemented, or implemented incorrectly.
  // Omit property instead of failing.
 }
 return state;
}

代码示例来源:origin: org.apache.gobblin/gobblin-core

protected String getPartitionPath(State properties) {
 if (this.partition.isPresent()) {
  boolean includePartitionerFieldNames = properties.getPropAsBoolean(ForkOperatorUtils
    .getPropertyNameForBranch(WRITER_INCLUDE_PARTITION_IN_FILE_NAMES, this.branches, this.branch), false);
  boolean removePathSeparators = properties.getPropAsBoolean(ForkOperatorUtils
    .getPropertyNameForBranch(WRITER_REPLACE_PATH_SEPARATORS_IN_PARTITIONS, this.branches, this.branch), false);
  return AvroUtils.serializeAsPath(this.partition.get(), includePartitionerFieldNames, removePathSeparators).toString();
 } else {
  return null;
 }
}

代码示例来源:origin: org.apache.gobblin/gobblin-core

@Override
public State getFinalState() {
 State state = new State();
 try {
  for (Map.Entry<GenericRecord, DataWriter<D>> entry : this.partitionWriters.asMap().entrySet()) {
   if (entry.getValue() instanceof FinalState) {
    State partitionFinalState = ((FinalState) entry.getValue()).getFinalState();
    if (this.shouldPartition) {
     for (String key : partitionFinalState.getPropertyNames()) {
      // Prevent overwriting final state across writers
      partitionFinalState.setProp(key + "_" + AvroUtils.serializeAsPath(entry.getKey(), false, true),
        partitionFinalState.getProp(key));
     }
    }
    state.addAll(partitionFinalState);
   }
  }
  state.setProp("RecordsWritten", recordsWritten());
  state.setProp("BytesWritten", bytesWritten());
 } catch (Exception exception) {
  log.warn("Failed to get final state." + exception.getMessage());
  // If Writer fails to return bytesWritten, it might not be implemented, or implemented incorrectly.
  // Omit property instead of failing.
 }
 return state;
}

相关文章