cascading.pipe.Pipe.getStepConfigDef()方法的使用及代码示例

x33g5p2x  于2022-01-26 转载在 其他  
字(8.8k)|赞(0)|评价(0)|浏览(112)

本文整理了Java中cascading.pipe.Pipe.getStepConfigDef()方法的一些代码示例，展示了Pipe.getStepConfigDef()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。Pipe.getStepConfigDef()方法的具体详情如下：
包路径：cascading.pipe.Pipe
类名称：Pipe
方法名：getStepConfigDef

Pipe.getStepConfigDef介绍

[英]Returns a ConfigDef instance that allows for process level properties to be set and made available via a resulting cascading.flow.FlowProcess instance when the pipe is invoked.

Any properties set on the stepConfigDef will not show up in any Flow configuration, but will show up in the current process cascading.flow.FlowStep (in Hadoop the MapReduce jobconf). Any value set in the stepConfigDef will be overridden by the pipe local #getConfigDef instance.

Use this method to tweak properties in the process step this pipe instance is planned into. In the case of the Hadoop platform, when set on a GroupBy instance, the number of reducers can be modified.
[中]返回一个ConfigDef实例，该实例允许设置进程级属性，并通过结果级联使其可用。流调用管道时的FlowProcess实例。
stepConfigDef上设置的任何属性都不会显示在任何流配置中，但会显示在当前流程级联中。流FlowStep（Hadoop中的MapReduce jobconf）。stepConfigDef中设置的任何值都将被管道本地#getConfigDef实例覆盖。
使用此方法可以在计划此管道实例的流程步骤中调整特性。在Hadoop平台中，当在GroupBy实例上设置时，可以修改缩减器的数量。

代码示例

代码示例来源：origin: LiveRamp/cascading_ext

private Pipe getBloomFilterPipe(Pipe largePipe, Fields largeJoinFields, Pipe smallPipe, Fields smallJoinFields) throws IOException {
 String bloomJobID = UUID.randomUUID().toString();
 Path bloomTempDir = FileSystemHelper.getRandomTemporaryPath("/tmp/bloom_tmp/");
 String bloomPartsDir = bloomTempDir + "/parts";
 String bloomFinalFilter = bloomTempDir + "/filter.bloomfilter";
 String approxCountPartsDir = bloomTempDir + "/approx_distinct_keys_parts/";
 Pipe filterPipe;
 smallPipe = new Each(smallPipe, smallJoinFields, new GetSerializedTuple());
 smallPipe = new CreateBloomFilter(smallPipe, bloomJobID, approxCountPartsDir, bloomPartsDir, "serialized-tuple-key");
 // This is a bit of a hack to:
 //  1) Force a dependency on the operations performed on RHS above (can't continue until they're done)
 //  2) Bind RHS to the flow, which wouldn't happen otherwise.
 // Note that RHS has no output, so there shouldn't be any danger in doing this.
 filterPipe = new NaiveMerge(largePipe.getName(), largePipe, smallPipe);
 // Load the bloom filter into memory and apply it to the LHS.
 filterPipe = new Each(filterPipe, largeJoinFields, new BloomJoinFilter(bloomJobID, false));
 ConfigDef config = filterPipe.getStepConfigDef();  // tell BloomAssemblyStrategy which bloom filter to expect
 config.setProperty(BloomProps.SOURCE_BLOOM_FILTER_ID, bloomJobID);
 config.setProperty(BloomProps.REQUIRED_BLOOM_FILTER_PATH, bloomFinalFilter);
 return filterPipe;
}

代码示例来源：origin: cwensel/cascading

@Test
public void testSubAssemblyConfigDef() throws IOException
 {
 getPlatform().copyFromLocal( inputFileNums20 );
 Tap source = getPlatform().getTextFile( new Fields( "line" ), inputFileNums20 );
 Pipe pipe = new Pipe( "test" );
 pipe = new ConfigSubAssembly( pipe, getPlatform().isDAG() );
 pipe.getConfigDef().setProperty( Mode.DEFAULT, "default", "pipe-default" );
 // steps on above value
 pipe.getStepConfigDef().setProperty( Mode.DEFAULT, "default", "process-default" );
 pipe.getConfigDef().setProperty( Mode.DEFAULT, "replace", "pipe-default" );
 pipe.getConfigDef().setProperty( Mode.REPLACE, "replace", "pipe-replace" );
 pipe.getNodeConfigDef().setProperty( Mode.REPLACE, "default-node", "node-replace" );
 pipe.getStepConfigDef().setProperty( Mode.DEFAULT, "replace", "process-default" );
 pipe.getStepConfigDef().setProperty( Mode.REPLACE, "replace", "process-replace" );
 pipe.getStepConfigDef().setProperty( Mode.DEFAULT, "default-node", "process-default" );
 Tap sink = getPlatform().getTextFile( getOutputPath( "subassembly-configdef" ), SinkMode.REPLACE );
 Flow flow = getPlatform().getFlowConnector().connect( source, sink, pipe );
 flow.complete();
 assertTrue( flow.resourceExists( sink ) );
 }
}

代码示例来源：origin: cascading/cascading-platform

@Test
public void testSubAssemblyConfigDef() throws IOException
 {
 getPlatform().copyFromLocal( inputFileNums20 );
 Tap source = getPlatform().getTextFile( new Fields( "line" ), inputFileNums20 );
 Pipe pipe = new Pipe( "test" );
 pipe = new ConfigSubAssembly( pipe, getPlatform().isDAG() );
 pipe.getConfigDef().setProperty( Mode.DEFAULT, "default", "pipe-default" );
 // steps on above value
 pipe.getStepConfigDef().setProperty( Mode.DEFAULT, "default", "process-default" );
 pipe.getConfigDef().setProperty( Mode.DEFAULT, "replace", "pipe-default" );
 pipe.getConfigDef().setProperty( Mode.REPLACE, "replace", "pipe-replace" );
 pipe.getNodeConfigDef().setProperty( Mode.REPLACE, "default-node", "node-replace" );
 pipe.getStepConfigDef().setProperty( Mode.DEFAULT, "replace", "process-default" );
 pipe.getStepConfigDef().setProperty( Mode.REPLACE, "replace", "process-replace" );
 pipe.getStepConfigDef().setProperty( Mode.DEFAULT, "default-node", "process-default" );
 Tap sink = getPlatform().getTextFile( getOutputPath( "subassembly-configdef" ), SinkMode.REPLACE );
 Flow flow = getPlatform().getFlowConnector().connect( source, sink, pipe );
 flow.complete();
 assertTrue( flow.resourceExists( sink ) );
 }
}

代码示例来源：origin: cwensel/cascading

@Test
public void testPipeConfigDef() throws IOException
 {
 getPlatform().copyFromLocal( inputFileNums20 );
 Tap source = getPlatform().getTextFile( new Fields( "line" ), inputFileNums20 );
 Pipe pipe = new Pipe( "test" );
 pipe = new Each( pipe, new IterateInsert( new Fields( "value" ), getPlatform().isDAG() ), Fields.ALL );
 pipe.getConfigDef().setProperty( Mode.DEFAULT, "default", "pipe-default" );
 // steps on above value
 pipe.getStepConfigDef().setProperty( Mode.DEFAULT, "default", "process-default" );
 pipe.getConfigDef().setProperty( Mode.DEFAULT, "replace", "pipe-default" );
 pipe.getConfigDef().setProperty( Mode.REPLACE, "replace", "pipe-replace" );
 pipe.getNodeConfigDef().setProperty( Mode.REPLACE, "default-node", "node-replace" );
 pipe.getStepConfigDef().setProperty( Mode.DEFAULT, "replace", "process-default" );
 pipe.getStepConfigDef().setProperty( Mode.REPLACE, "replace", "process-replace" );
 pipe.getStepConfigDef().setProperty( Mode.DEFAULT, "default-node", "process-default" );
 Tap sink = getPlatform().getTextFile( getOutputPath( "configdef" ), SinkMode.REPLACE );
 Flow flow = getPlatform().getFlowConnector().connect( source, sink, pipe );
 flow.complete();
 assertTrue( flow.resourceExists( sink ) );
 }

代码示例来源：origin: LiveRamp/cascading_ext

public CreateBloomFilter(Pipe keys, String bloomFilterID, String approxCountPartsDir, String bloomPartsDir, String keyBytesField, HashFunctionFactory hashFactory) throws IOException {
 super(keys);
 Pipe smallPipe = new Each(keys, new Fields(keyBytesField), new GetIndices(hashFactory), new Fields("split", "index", "hash_num"));
 smallPipe = new Each(smallPipe, new Fields("split", "index", "hash_num"), new Unique.FilterPartialDuplicates());
 smallPipe = new GroupBy(smallPipe, new Fields("split"));
 smallPipe = new Every(smallPipe, new Fields("index", "hash_num"), new CreateBloomFilterFromIndices(), Fields.ALL);
 ConfigDef bloomDef = smallPipe.getStepConfigDef();
 bloomDef.setProperty(BloomProps.BLOOM_FILTER_PARTS_DIR, bloomPartsDir);
 bloomDef.setProperty(BloomProps.BLOOM_KEYS_COUNTS_DIR, approxCountPartsDir);
 bloomDef.setProperty(BloomProps.TARGET_BLOOM_FILTER_ID, bloomFilterID);
 setTails(smallPipe);
}

代码示例来源：origin: cascading/cascading-platform

@Test
public void testPipeConfigDef() throws IOException
 {
 getPlatform().copyFromLocal( inputFileNums20 );
 Tap source = getPlatform().getTextFile( new Fields( "line" ), inputFileNums20 );
 Pipe pipe = new Pipe( "test" );
 pipe = new Each( pipe, new IterateInsert( new Fields( "value" ), getPlatform().isDAG() ), Fields.ALL );
 pipe.getConfigDef().setProperty( Mode.DEFAULT, "default", "pipe-default" );
 // steps on above value
 pipe.getStepConfigDef().setProperty( Mode.DEFAULT, "default", "process-default" );
 pipe.getConfigDef().setProperty( Mode.DEFAULT, "replace", "pipe-default" );
 pipe.getConfigDef().setProperty( Mode.REPLACE, "replace", "pipe-replace" );
 pipe.getNodeConfigDef().setProperty( Mode.REPLACE, "default-node", "node-replace" );
 pipe.getStepConfigDef().setProperty( Mode.DEFAULT, "replace", "process-default" );
 pipe.getStepConfigDef().setProperty( Mode.REPLACE, "replace", "process-replace" );
 pipe.getStepConfigDef().setProperty( Mode.DEFAULT, "default-node", "process-default" );
 Tap sink = getPlatform().getTextFile( getOutputPath( "configdef" ), SinkMode.REPLACE );
 Flow flow = getPlatform().getFlowConnector().connect( source, sink, pipe );
 flow.complete();
 assertTrue( flow.resourceExists( sink ) );
 }

内容来源于网络，如有侵权，请联系作者删除！

Pipe

相关文章

    查看更多
    微信公众号

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com