cascading.flow.Flow.setFlowStepStrategy()方法的使用及代码示例

x33g5p2x  于2022-01-19 转载在 其他  
字(4.3k)|赞(0)|评价(0)|浏览(109)

本文整理了Java中cascading.flow.Flow.setFlowStepStrategy()方法的一些代码示例,展示了Flow.setFlowStepStrategy()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。Flow.setFlowStepStrategy()方法的具体详情如下:
包路径:cascading.flow.Flow
类名称:Flow
方法名:setFlowStepStrategy

Flow.setFlowStepStrategy介绍

[英]Sets a default FlowStepStrategy instance.

Use a FlowStepStrategy to change cascading.flow.FlowStep configuration properties before the properties are submitted to the underlying platform for the step unit of work.
[中]设置默认的FlowStepStrategy实例。
使用FlowStep策略更改级联。流将属性提交到step工作单元的基础平台之前的FlowStep配置属性。

代码示例

代码示例来源:origin: org.springframework.data/spring-cascading

@Override
public void afterPropertiesSet() throws Exception {
  flow = createFlow();
  if (skipStrategy != null) {
    flow.setFlowSkipStrategy(skipStrategy);
  }
  if (stepStrategy != null) {
    flow.setFlowStepStrategy(stepStrategy);
  }
  if (listeners != null) {
    for (FlowListener listener : listeners) {
      flow.addListener(listener);
    }
  }
  if (StringUtils.hasText(writeDOT)) {
    flow.writeDOT(writeDOT);
  }
  if (StringUtils.hasText(writeStepsDOT)) {
    flow.writeStepsDOT(writeStepsDOT);
  }
  if (priority != null) {
    flow.setSubmitPriority(priority);
  }
}

代码示例来源:origin: LiveRamp/cascading_ext

public static void main(String[] args) throws IOException {
 if (args.length != 1) {
  System.out.println("Usage: hadoop jar cascading_ext.job.jar com.liveramp.cascading_ext.example.BloomJoinExampleWithoutCascadingUtil <output dir>");
  return;
 }
 String outputDir = args[0];
 Hfs sink = new Hfs(new SequenceFile(new Fields("field1", "field2", "field3", "field4")), outputDir);
 Pipe source1 = new Pipe("source1");
 Pipe source2 = new Pipe("source2");
 Pipe joined = new BloomJoin(source1, new Fields("field1"),
   source2, new Fields("field3"));
 Map<String, Tap> sources = new HashMap<String, Tap>();
 sources.put("source1", ExampleFixtures.SOURCE_TAP_1);
 sources.put("source2", ExampleFixtures.SOURCE_TAP_2);
 //  set some default properties and set the flow step strategy
 Flow f = new HadoopFlowConnector(BloomProps.getDefaultProperties()).connect("Example BloomJoin", sources, sink, joined);
 f.setFlowStepStrategy(new BloomAssemblyStrategy());
 f.complete();
 //  Take a look at the output tuples
 TupleEntryIterator output = sink.openForRead(CascadingUtil.get().getFlowProcess());
 System.out.println("Output tuples from flow:");
 while (output.hasNext()) {
  System.out.println(output.next().getTuple());
 }
}

代码示例来源:origin: cwensel/cascading

@Test
public void testFlowStepStrategy() throws Exception
 {
 getPlatform().copyFromLocal( inputFileApache );
 Tap source = getPlatform().getTextFile( new Fields( "offset", "line" ), inputFileApache );
 Pipe pipe = new Pipe( "test" );
 pipe = new Each( pipe, new Fields( "line" ), new RegexParser( new Fields( "ip" ), "^[^ ]*" ), new Fields( "ip" ) );
 pipe = new GroupBy( pipe, new Fields( "ip" ) );
 pipe = new Every( pipe, new Count(), new Fields( "ip", "count" ) );
 Tap sink = getPlatform().getTextFile( getOutputPath( "simple" ), SinkMode.REPLACE );
 Flow flow = getPlatform().getFlowConnector().connect( source, sink, pipe );
 final boolean[] wasApplied = {false};
 flow.setFlowStepStrategy( new FlowStepStrategy()
  {
  @Override
  public void apply( Flow flow, List predecessorSteps, FlowStep flowStep )
   {
   wasApplied[ 0 ] = true;
   assertTrue( predecessorSteps.isEmpty() );
   }
  } );
 flow.complete();
 assertTrue( wasApplied[ 0 ] );
 validateLength( flow, 8, null );
 }

代码示例来源:origin: cascading/cascading-platform

@Test
public void testFlowStepStrategy() throws Exception
 {
 getPlatform().copyFromLocal( inputFileApache );
 Tap source = getPlatform().getTextFile( new Fields( "offset", "line" ), inputFileApache );
 Pipe pipe = new Pipe( "test" );
 pipe = new Each( pipe, new Fields( "line" ), new RegexParser( new Fields( "ip" ), "^[^ ]*" ), new Fields( "ip" ) );
 pipe = new GroupBy( pipe, new Fields( "ip" ) );
 pipe = new Every( pipe, new Count(), new Fields( "ip", "count" ) );
 Tap sink = getPlatform().getTextFile( getOutputPath( "simple" ), SinkMode.REPLACE );
 Flow flow = getPlatform().getFlowConnector().connect( source, sink, pipe );
 final boolean[] wasApplied = {false};
 flow.setFlowStepStrategy( new FlowStepStrategy()
  {
  @Override
  public void apply( Flow flow, List predecessorSteps, FlowStep flowStep )
   {
   wasApplied[ 0 ] = true;
   assertTrue( predecessorSteps.isEmpty() );
   }
  } );
 flow.complete();
 assertTrue( wasApplied[ 0 ] );
 validateLength( flow, 8, null );
 }

相关文章