cascading.pipe.Pipe.getNodeConfigDef()方法的使用及代码示例

x33g5p2x  于2022-01-26 转载在 其他  
字(6.5k)|赞(0)|评价(0)|浏览(117)

本文整理了Java中cascading.pipe.Pipe.getNodeConfigDef()方法的一些代码示例,展示了Pipe.getNodeConfigDef()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。Pipe.getNodeConfigDef()方法的具体详情如下:
包路径:cascading.pipe.Pipe
类名称:Pipe
方法名:getNodeConfigDef

Pipe.getNodeConfigDef介绍

[英]Returns a ConfigDef instance that allows for process level properties to be set and made available via a resulting cascading.flow.FlowProcess instance when the pipe is invoked.

Any properties set on the nodeConfigDef will not show up in any Flow configuration, but will show up in the current process cascading.flow.FlowNode (in Apache Tez the Vertex configuration). Any value set in the nodeConfigDef will be overridden by the pipe local #getConfigDef instance.

Use this method to tweak properties in the process node this pipe instance is planned into. In the case of the Apache Tez platform, when set on a GroupBy instance, the number of gather partitions can be modified.

In the case of any Pipe that spans FlowNode boundaries, like GroupBy and CoGroup may on some platforms, any ConfigDef properties will be applied to the downstream FlowNode. That is, if a GroupBy is the source to a node, any node ConfigDef properties will be applied. If the GroupBy encountered when applying properties is on the sink side of a node, the properties will be ignored.
[中]

代码示例

代码示例来源:origin: cwensel/cascading

@Test
public void testSubAssemblyConfigDef() throws IOException
 {
 getPlatform().copyFromLocal( inputFileNums20 );
 Tap source = getPlatform().getTextFile( new Fields( "line" ), inputFileNums20 );
 Pipe pipe = new Pipe( "test" );
 pipe = new ConfigSubAssembly( pipe, getPlatform().isDAG() );
 pipe.getConfigDef().setProperty( Mode.DEFAULT, "default", "pipe-default" );
 // steps on above value
 pipe.getStepConfigDef().setProperty( Mode.DEFAULT, "default", "process-default" );
 pipe.getConfigDef().setProperty( Mode.DEFAULT, "replace", "pipe-default" );
 pipe.getConfigDef().setProperty( Mode.REPLACE, "replace", "pipe-replace" );
 pipe.getNodeConfigDef().setProperty( Mode.REPLACE, "default-node", "node-replace" );
 pipe.getStepConfigDef().setProperty( Mode.DEFAULT, "replace", "process-default" );
 pipe.getStepConfigDef().setProperty( Mode.REPLACE, "replace", "process-replace" );
 pipe.getStepConfigDef().setProperty( Mode.DEFAULT, "default-node", "process-default" );
 Tap sink = getPlatform().getTextFile( getOutputPath( "subassembly-configdef" ), SinkMode.REPLACE );
 Flow flow = getPlatform().getFlowConnector().connect( source, sink, pipe );
 flow.complete();
 assertTrue( flow.resourceExists( sink ) );
 }
}

代码示例来源:origin: cascading/cascading-platform

@Test
public void testSubAssemblyConfigDef() throws IOException
 {
 getPlatform().copyFromLocal( inputFileNums20 );
 Tap source = getPlatform().getTextFile( new Fields( "line" ), inputFileNums20 );
 Pipe pipe = new Pipe( "test" );
 pipe = new ConfigSubAssembly( pipe, getPlatform().isDAG() );
 pipe.getConfigDef().setProperty( Mode.DEFAULT, "default", "pipe-default" );
 // steps on above value
 pipe.getStepConfigDef().setProperty( Mode.DEFAULT, "default", "process-default" );
 pipe.getConfigDef().setProperty( Mode.DEFAULT, "replace", "pipe-default" );
 pipe.getConfigDef().setProperty( Mode.REPLACE, "replace", "pipe-replace" );
 pipe.getNodeConfigDef().setProperty( Mode.REPLACE, "default-node", "node-replace" );
 pipe.getStepConfigDef().setProperty( Mode.DEFAULT, "replace", "process-default" );
 pipe.getStepConfigDef().setProperty( Mode.REPLACE, "replace", "process-replace" );
 pipe.getStepConfigDef().setProperty( Mode.DEFAULT, "default-node", "process-default" );
 Tap sink = getPlatform().getTextFile( getOutputPath( "subassembly-configdef" ), SinkMode.REPLACE );
 Flow flow = getPlatform().getFlowConnector().connect( source, sink, pipe );
 flow.complete();
 assertTrue( flow.resourceExists( sink ) );
 }
}

代码示例来源:origin: cwensel/cascading

@Test
public void testPipeConfigDef() throws IOException
 {
 getPlatform().copyFromLocal( inputFileNums20 );
 Tap source = getPlatform().getTextFile( new Fields( "line" ), inputFileNums20 );
 Pipe pipe = new Pipe( "test" );
 pipe = new Each( pipe, new IterateInsert( new Fields( "value" ), getPlatform().isDAG() ), Fields.ALL );
 pipe.getConfigDef().setProperty( Mode.DEFAULT, "default", "pipe-default" );
 // steps on above value
 pipe.getStepConfigDef().setProperty( Mode.DEFAULT, "default", "process-default" );
 pipe.getConfigDef().setProperty( Mode.DEFAULT, "replace", "pipe-default" );
 pipe.getConfigDef().setProperty( Mode.REPLACE, "replace", "pipe-replace" );
 pipe.getNodeConfigDef().setProperty( Mode.REPLACE, "default-node", "node-replace" );
 pipe.getStepConfigDef().setProperty( Mode.DEFAULT, "replace", "process-default" );
 pipe.getStepConfigDef().setProperty( Mode.REPLACE, "replace", "process-replace" );
 pipe.getStepConfigDef().setProperty( Mode.DEFAULT, "default-node", "process-default" );
 Tap sink = getPlatform().getTextFile( getOutputPath( "configdef" ), SinkMode.REPLACE );
 Flow flow = getPlatform().getFlowConnector().connect( source, sink, pipe );
 flow.complete();
 assertTrue( flow.resourceExists( sink ) );
 }

代码示例来源:origin: cascading/cascading-platform

@Test
public void testPipeConfigDef() throws IOException
 {
 getPlatform().copyFromLocal( inputFileNums20 );
 Tap source = getPlatform().getTextFile( new Fields( "line" ), inputFileNums20 );
 Pipe pipe = new Pipe( "test" );
 pipe = new Each( pipe, new IterateInsert( new Fields( "value" ), getPlatform().isDAG() ), Fields.ALL );
 pipe.getConfigDef().setProperty( Mode.DEFAULT, "default", "pipe-default" );
 // steps on above value
 pipe.getStepConfigDef().setProperty( Mode.DEFAULT, "default", "process-default" );
 pipe.getConfigDef().setProperty( Mode.DEFAULT, "replace", "pipe-default" );
 pipe.getConfigDef().setProperty( Mode.REPLACE, "replace", "pipe-replace" );
 pipe.getNodeConfigDef().setProperty( Mode.REPLACE, "default-node", "node-replace" );
 pipe.getStepConfigDef().setProperty( Mode.DEFAULT, "replace", "process-default" );
 pipe.getStepConfigDef().setProperty( Mode.REPLACE, "replace", "process-replace" );
 pipe.getStepConfigDef().setProperty( Mode.DEFAULT, "default-node", "process-default" );
 Tap sink = getPlatform().getTextFile( getOutputPath( "configdef" ), SinkMode.REPLACE );
 Flow flow = getPlatform().getFlowConnector().connect( source, sink, pipe );
 flow.complete();
 assertTrue( flow.resourceExists( sink ) );
 }

相关文章