本文整理了Java中org.apache.gobblin.util.AvroUtils.serializeAsPath()
方法的一些代码示例,展示了AvroUtils.serializeAsPath()
的具体用法。这些代码示例主要来源于Github
/Stackoverflow
/Maven
等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。AvroUtils.serializeAsPath()
方法的具体详情如下:
包路径:org.apache.gobblin.util.AvroUtils
类名称:AvroUtils
方法名:serializeAsPath
[英]Serialize a generic record as a relative Path. Useful for converting GenericRecord type keys into file system locations. For example {field1=v1, field2=v2} returns field1=v1/field2=v2 if includeFieldNames is true, or v1/v2 if it is false. Illegal HDFS tokens such as ':' and '\' will be replaced with ''. Additionally, parameter replacePathSeparators controls whether to replace path separators ('/') with ''.
[中]将泛型记录序列化为相对路径。用于将GenericRecord类型密钥转换为文件系统位置。例如{field1=v1,field2=v2}如果includeFieldNames为true,则返回field1=v1/field2=v2;如果includeFieldNames为false,则返回v1/v2。非法的HDFS令牌,如“:”和“\”将被替换为“\”。此外,参数replacePathSeparators控制是否将路径分隔符(“/”)替换为“\u1”。
代码示例来源:origin: apache/incubator-gobblin
protected String getPartitionPath(State properties) {
if (this.partition.isPresent()) {
boolean includePartitionerFieldNames = properties.getPropAsBoolean(ForkOperatorUtils
.getPropertyNameForBranch(WRITER_INCLUDE_PARTITION_IN_FILE_NAMES, this.branches, this.branch), false);
boolean removePathSeparators = properties.getPropAsBoolean(ForkOperatorUtils
.getPropertyNameForBranch(WRITER_REPLACE_PATH_SEPARATORS_IN_PARTITIONS, this.branches, this.branch), false);
return AvroUtils.serializeAsPath(this.partition.get(), includePartitionerFieldNames, removePathSeparators).toString();
} else {
return null;
}
}
代码示例来源:origin: apache/incubator-gobblin
@Test public void testSerializeAsPath() throws Exception {
Schema schema =
new Schema.Parser().parse("{\"type\":\"record\", \"name\":\"test\", " + "\"fields\":["
+ "{\"name\": \"name\", \"type\": \"string\"}, " + "{\"name\": \"title\", \"type\": \"string\"}" + "]}");
GenericRecord partition = new GenericData.Record(schema);
partition.put("name", "a/b:c\\d e");
partition.put("title", "title");
Assert.assertEquals(AvroUtils.serializeAsPath(partition, true, true), new Path("name=a_b_c_d_e/title=title"));
Assert.assertEquals(AvroUtils.serializeAsPath(partition, false, true), new Path("a_b_c_d_e/title"));
Assert.assertEquals(AvroUtils.serializeAsPath(partition, false, false), new Path("a/b_c_d_e/title"));
}
代码示例来源:origin: apache/incubator-gobblin
@Override
public State getFinalState() {
State state = new State();
try {
for (Map.Entry<GenericRecord, DataWriter<D>> entry : this.partitionWriters.asMap().entrySet()) {
if (entry.getValue() instanceof FinalState) {
State partitionFinalState = ((FinalState) entry.getValue()).getFinalState();
if (this.shouldPartition) {
for (String key : partitionFinalState.getPropertyNames()) {
// Prevent overwriting final state across writers
partitionFinalState.setProp(key + "_" + AvroUtils.serializeAsPath(entry.getKey(), false, true),
partitionFinalState.getProp(key));
}
}
state.addAll(partitionFinalState);
}
}
state.setProp("RecordsWritten", recordsWritten());
state.setProp("BytesWritten", bytesWritten());
} catch (Exception exception) {
log.warn("Failed to get final state." + exception.getMessage());
// If Writer fails to return bytesWritten, it might not be implemented, or implemented incorrectly.
// Omit property instead of failing.
}
return state;
}
代码示例来源:origin: org.apache.gobblin/gobblin-core
protected String getPartitionPath(State properties) {
if (this.partition.isPresent()) {
boolean includePartitionerFieldNames = properties.getPropAsBoolean(ForkOperatorUtils
.getPropertyNameForBranch(WRITER_INCLUDE_PARTITION_IN_FILE_NAMES, this.branches, this.branch), false);
boolean removePathSeparators = properties.getPropAsBoolean(ForkOperatorUtils
.getPropertyNameForBranch(WRITER_REPLACE_PATH_SEPARATORS_IN_PARTITIONS, this.branches, this.branch), false);
return AvroUtils.serializeAsPath(this.partition.get(), includePartitionerFieldNames, removePathSeparators).toString();
} else {
return null;
}
}
代码示例来源:origin: org.apache.gobblin/gobblin-core
@Override
public State getFinalState() {
State state = new State();
try {
for (Map.Entry<GenericRecord, DataWriter<D>> entry : this.partitionWriters.asMap().entrySet()) {
if (entry.getValue() instanceof FinalState) {
State partitionFinalState = ((FinalState) entry.getValue()).getFinalState();
if (this.shouldPartition) {
for (String key : partitionFinalState.getPropertyNames()) {
// Prevent overwriting final state across writers
partitionFinalState.setProp(key + "_" + AvroUtils.serializeAsPath(entry.getKey(), false, true),
partitionFinalState.getProp(key));
}
}
state.addAll(partitionFinalState);
}
}
state.setProp("RecordsWritten", recordsWritten());
state.setProp("BytesWritten", bytesWritten());
} catch (Exception exception) {
log.warn("Failed to get final state." + exception.getMessage());
// If Writer fails to return bytesWritten, it might not be implemented, or implemented incorrectly.
// Omit property instead of failing.
}
return state;
}
内容来源于网络,如有侵权,请联系作者删除!