org.apache.gobblin.util.WriterUtils.getDataPublisherFinalDir()方法的使用及代码示例

x33g5p2x  于2022-02-03 转载在 其他  
字(3.7k)|赞(0)|评价(0)|浏览(116)

本文整理了Java中org.apache.gobblin.util.WriterUtils.getDataPublisherFinalDir()方法的一些代码示例,展示了WriterUtils.getDataPublisherFinalDir()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。WriterUtils.getDataPublisherFinalDir()方法的具体详情如下:
包路径:org.apache.gobblin.util.WriterUtils
类名称:WriterUtils
方法名:getDataPublisherFinalDir

WriterUtils.getDataPublisherFinalDir介绍

[英]Get the Path corresponding the to the directory a given org.apache.gobblin.publisher.BaseDataPublisher should commits its output data. The final output data directory is determined by combining the ConfigurationKeys#DATA_PUBLISHER_FINAL_DIR and the ConfigurationKeys#WRITER_FILE_PATH.
[中]获取与给定组织的目录对应的路径。阿帕奇。戈布林。出版商BaseDataPublisher应提交其输出数据。最终输出数据目录是通过组合ConfigurationKeys#data _PUBLISHER _final _DIR和ConfigurationKeys#WRITER _FILE _路径确定的。

代码示例

代码示例来源:origin: apache/incubator-gobblin

/**
 * Get the output directory path this {@link BaseDataPublisher} will write to.
 *
 * <p>
 *   This is the default implementation. Subclasses of {@link BaseDataPublisher} may override this
 *   to write to a custom directory or write using a custom directory structure or naming pattern.
 * </p>
 *
 * @param workUnitState a {@link WorkUnitState} object
 * @param branchId the fork branch ID
 * @return the output directory path this {@link BaseDataPublisher} will write to
 */
protected Path getPublisherOutputDir(WorkUnitState workUnitState, int branchId) {
 return WriterUtils.getDataPublisherFinalDir(workUnitState, this.numBranches, branchId);
}

代码示例来源:origin: apache/incubator-gobblin

@Test
public void testGetDataPublisherFinalOutputDir() {
 State state = new State();
 state.setProp(ConfigurationKeys.DATA_PUBLISHER_FINAL_DIR, TEST_DATA_PUBLISHER_FINAL_DIR);
 state.setProp(ConfigurationKeys.WRITER_FILE_PATH, TEST_WRITER_FILE_PATH);
 Assert.assertEquals(WriterUtils.getDataPublisherFinalDir(state, 0, 0), new Path(TEST_DATA_PUBLISHER_FINAL_DIR,
   TEST_WRITER_FILE_PATH));
 state.setProp(ConfigurationKeys.DATA_PUBLISHER_FINAL_DIR + ".0", TEST_DATA_PUBLISHER_FINAL_DIR);
 state.setProp(ConfigurationKeys.WRITER_FILE_PATH + ".0", TEST_WRITER_FILE_PATH);
 Assert.assertEquals(WriterUtils.getDataPublisherFinalDir(state, 2, 0), new Path(TEST_DATA_PUBLISHER_FINAL_DIR,
   TEST_WRITER_FILE_PATH));
 state.setProp(ConfigurationKeys.DATA_PUBLISHER_FINAL_DIR + ".1", TEST_DATA_PUBLISHER_FINAL_DIR);
 state.setProp(ConfigurationKeys.WRITER_FILE_PATH + ".1", TEST_WRITER_FILE_PATH);
 Assert.assertEquals(WriterUtils.getDataPublisherFinalDir(state, 2, 1), new Path(TEST_DATA_PUBLISHER_FINAL_DIR,
   TEST_WRITER_FILE_PATH));
}

代码示例来源:origin: apache/incubator-gobblin

+ "is not specified. Trying to get the orignal schema from previous avro files.");
originalSchemaPath = WriterUtils
  .getDataPublisherFinalDir(workUnitState, workUnitState.getPropAsInt(ConfigurationKeys.FORK_BRANCHES_KEY, 1),
    workUnitState.getPropAsInt(ConfigurationKeys.FORK_BRANCH_ID_KEY, 0)).getParent();

代码示例来源:origin: org.apache.gobblin/gobblin-core

/**
 * Get the output directory path this {@link BaseDataPublisher} will write to.
 *
 * <p>
 *   This is the default implementation. Subclasses of {@link BaseDataPublisher} may override this
 *   to write to a custom directory or write using a custom directory structure or naming pattern.
 * </p>
 *
 * @param workUnitState a {@link WorkUnitState} object
 * @param branchId the fork branch ID
 * @return the output directory path this {@link BaseDataPublisher} will write to
 */
protected Path getPublisherOutputDir(WorkUnitState workUnitState, int branchId) {
 return WriterUtils.getDataPublisherFinalDir(workUnitState, this.numBranches, branchId);
}

代码示例来源:origin: org.apache.gobblin/gobblin-core

+ "is not specified. Trying to get the orignal schema from previous avro files.");
originalSchemaPath = WriterUtils
  .getDataPublisherFinalDir(workUnitState, workUnitState.getPropAsInt(ConfigurationKeys.FORK_BRANCHES_KEY, 1),
    workUnitState.getPropAsInt(ConfigurationKeys.FORK_BRANCH_ID_KEY, 0)).getParent();

相关文章