org.apache.flink.api.java.operators.GroupReduceOperator.output()方法的使用及代码示例

x33g5p2x  于2022-01-20 转载在 其他  
字(11.8k)|赞(0)|评价(0)|浏览(108)

本文整理了Java中org.apache.flink.api.java.operators.GroupReduceOperator.output()方法的一些代码示例，展示了GroupReduceOperator.output()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。GroupReduceOperator.output()方法的具体详情如下：
包路径：org.apache.flink.api.java.operators.GroupReduceOperator
类名称：GroupReduceOperator
方法名：output

GroupReduceOperator.output介绍

暂无

代码示例

代码示例来源：origin: apache/flink

@Test
public void testPartitionCustomOperatorPreservesFields() {
  try {
    ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
    
    DataSet<Tuple2<Long, Long>> data = env.fromCollection(Collections.singleton(new Tuple2<>(0L, 0L)));
    
    data.partitionCustom(new Partitioner<Long>() {
        public int partition(Long key, int numPartitions) { return key.intValue(); }
      }, 1)
      .groupBy(1)
      .reduceGroup(new IdentityGroupReducerCombinable<Tuple2<Long, Long>>())
      .output(new DiscardingOutputFormat<Tuple2<Long, Long>>());
    
    Plan p = env.createProgramPlan();
    OptimizedPlan op = compileNoStats(p);
    
    SinkPlanNode sink = op.getDataSinks().iterator().next();
    SingleInputPlanNode reducer = (SingleInputPlanNode) sink.getInput().getSource();
    SingleInputPlanNode partitioner = (SingleInputPlanNode) reducer.getInput().getSource();
    assertEquals(ShipStrategyType.FORWARD, reducer.getInput().getShipStrategy());
    assertEquals(ShipStrategyType.PARTITION_CUSTOM, partitioner.getInput().getShipStrategy());
  }
  catch (Exception e) {
    e.printStackTrace();
    fail(e.getMessage());
  }
}

代码示例来源：origin: apache/flink

.groupBy(1)
.reduceGroup(new IdentityGroupReducerCombinable<Tuple2<Long,Long>>())
.output(new DiscardingOutputFormat<Tuple2<Long, Long>>());

代码示例来源：origin: apache/flink

.reduceGroup((GroupReduceFunction<Tuple3<Double, StringValue, LongValue>, String>) (values, out) -> {})
.returns(String.class)
.output(new DiscardingOutputFormat<>());

代码示例来源：origin: apache/flink

public void reduce(Iterable<Double> values, Collector<Double> out) {}
}).name("reducer")
.output(new DiscardingOutputFormat<Double>()).name("sink");

代码示例来源：origin: apache/flink

@Test
public void testCustomPartitioningTupleGroupReduce() {
  try {
    ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
    
    DataSet<Tuple2<Integer, Integer>> data = env.fromElements(new Tuple2<Integer, Integer>(0, 0))
        .rebalance().setParallelism(4);
    
    data.groupBy(0).withPartitioner(new TestPartitionerInt())
      .reduceGroup(new IdentityGroupReducerCombinable<Tuple2<Integer,Integer>>())
      .output(new DiscardingOutputFormat<Tuple2<Integer, Integer>>());
    
    Plan p = env.createProgramPlan();
    OptimizedPlan op = compileNoStats(p);
    
    SinkPlanNode sink = op.getDataSinks().iterator().next();
    SingleInputPlanNode reducer = (SingleInputPlanNode) sink.getInput().getSource();
    SingleInputPlanNode combiner = (SingleInputPlanNode) reducer.getInput().getSource();
    
    assertEquals(ShipStrategyType.FORWARD, sink.getInput().getShipStrategy());
    assertEquals(ShipStrategyType.PARTITION_CUSTOM, reducer.getInput().getShipStrategy());
    assertEquals(ShipStrategyType.FORWARD, combiner.getInput().getShipStrategy());
  }
  catch (Exception e) {
    e.printStackTrace();
    fail(e.getMessage());
  }
}

代码示例来源：origin: apache/flink

@Test
public void testCustomPartitioningTupleGroupReduce() {
  try {
    ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
    DataSet<Pojo2> data = env.fromElements(new Pojo2())
        .rebalance().setParallelism(4);
    data.groupBy("a").withPartitioner(new TestPartitionerInt())
        .reduceGroup(new IdentityGroupReducerCombinable<Pojo2>())
        .output(new DiscardingOutputFormat<Pojo2>());
    Plan p = env.createProgramPlan();
    OptimizedPlan op = compileNoStats(p);
    SinkPlanNode sink = op.getDataSinks().iterator().next();
    SingleInputPlanNode reducer = (SingleInputPlanNode) sink.getInput().getSource();
    SingleInputPlanNode combiner = (SingleInputPlanNode) reducer.getInput().getSource();
    assertEquals(ShipStrategyType.FORWARD, sink.getInput().getShipStrategy());
    assertEquals(ShipStrategyType.PARTITION_CUSTOM, reducer.getInput().getShipStrategy());
    assertEquals(ShipStrategyType.FORWARD, combiner.getInput().getShipStrategy());
  }
  catch (Exception e) {
    e.printStackTrace();
    fail(e.getMessage());
  }
}

代码示例来源：origin: apache/flink

@Test
public void testCustomPartitioningKeySelectorGroupReduce() {
  try {
    ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
    
    DataSet<Tuple2<Integer, Integer>> data = env.fromElements(new Tuple2<Integer, Integer>(0, 0))
        .rebalance().setParallelism(4);
    
    data.groupBy(new TestKeySelector<Tuple2<Integer,Integer>>())
      .withPartitioner(new TestPartitionerInt())
        .reduceGroup(new IdentityGroupReducerCombinable<Tuple2<Integer,Integer>>())
      .output(new DiscardingOutputFormat<Tuple2<Integer, Integer>>());
    
    Plan p = env.createProgramPlan();
    OptimizedPlan op = compileNoStats(p);
    
    SinkPlanNode sink = op.getDataSinks().iterator().next();
    SingleInputPlanNode reducer = (SingleInputPlanNode) sink.getInput().getSource();
    SingleInputPlanNode combiner = (SingleInputPlanNode) reducer.getInput().getSource();
    
    assertEquals(ShipStrategyType.FORWARD, sink.getInput().getShipStrategy());
    assertEquals(ShipStrategyType.PARTITION_CUSTOM, reducer.getInput().getShipStrategy());
    assertEquals(ShipStrategyType.FORWARD, combiner.getInput().getShipStrategy());
  }
  catch (Exception e) {
    e.printStackTrace();
    fail(e.getMessage());
  }
}

代码示例来源：origin: apache/flink

@Test
public void testCustomPartitioningTupleGroupReduceSorted() {
  try {
    ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
    DataSet<Pojo3> data = env.fromElements(new Pojo3())
        .rebalance().setParallelism(4);
    data.groupBy("a").withPartitioner(new TestPartitionerInt())
        .sortGroup("b", Order.ASCENDING)
        .reduceGroup(new IdentityGroupReducerCombinable<Pojo3>())
        .output(new DiscardingOutputFormat<Pojo3>());
    Plan p = env.createProgramPlan();
    OptimizedPlan op = compileNoStats(p);
    SinkPlanNode sink = op.getDataSinks().iterator().next();
    SingleInputPlanNode reducer = (SingleInputPlanNode) sink.getInput().getSource();
    SingleInputPlanNode combiner = (SingleInputPlanNode) reducer.getInput().getSource();
    assertEquals(ShipStrategyType.FORWARD, sink.getInput().getShipStrategy());
    assertEquals(ShipStrategyType.PARTITION_CUSTOM, reducer.getInput().getShipStrategy());
    assertEquals(ShipStrategyType.FORWARD, combiner.getInput().getShipStrategy());
  }
  catch (Exception e) {
    e.printStackTrace();
    fail(e.getMessage());
  }
}

代码示例来源：origin: apache/flink

@Test
public void testCustomPartitioningTupleGroupReduceSorted() {
  try {
    ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
    
    DataSet<Tuple3<Integer, Integer, Integer>> data = env.fromElements(new Tuple3<Integer, Integer, Integer>(0, 0, 0))
        .rebalance().setParallelism(4);
    
    data.groupBy(0).withPartitioner(new TestPartitionerInt())
      .sortGroup(1, Order.ASCENDING)
      .reduceGroup(new IdentityGroupReducerCombinable<Tuple3<Integer,Integer,Integer>>())
      .output(new DiscardingOutputFormat<Tuple3<Integer, Integer, Integer>>());
    
    Plan p = env.createProgramPlan();
    OptimizedPlan op = compileNoStats(p);
    
    SinkPlanNode sink = op.getDataSinks().iterator().next();
    SingleInputPlanNode reducer = (SingleInputPlanNode) sink.getInput().getSource();
    SingleInputPlanNode combiner = (SingleInputPlanNode) reducer.getInput().getSource();
    
    assertEquals(ShipStrategyType.FORWARD, sink.getInput().getShipStrategy());
    assertEquals(ShipStrategyType.PARTITION_CUSTOM, reducer.getInput().getShipStrategy());
    assertEquals(ShipStrategyType.FORWARD, combiner.getInput().getShipStrategy());
  }
  catch (Exception e) {
    e.printStackTrace();
    fail(e.getMessage());
  }
}

代码示例来源：origin: apache/flink

.sortGroup(2, Order.DESCENDING)
.reduceGroup(new IdentityGroupReducerCombinable<Tuple4<Integer,Integer,Integer,Integer>>())
.output(new DiscardingOutputFormat<Tuple4<Integer, Integer, Integer, Integer>>());

代码示例来源：origin: apache/flink

.sortGroup(new TestKeySelector<Tuple3<Integer, Integer, Integer>>(), Order.ASCENDING)
.reduceGroup(new IdentityGroupReducerCombinable<Tuple3<Integer,Integer,Integer>>())
.output(new DiscardingOutputFormat<Tuple3<Integer, Integer, Integer>>());

代码示例来源：origin: apache/flink

@Test
public void testCustomPartitioningTupleGroupReduceSorted2() {
  try {
    ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
    DataSet<Pojo4> data = env.fromElements(new Pojo4())
        .rebalance().setParallelism(4);
    data.groupBy("a").withPartitioner(new TestPartitionerInt())
        .sortGroup("b", Order.ASCENDING)
        .sortGroup("c", Order.DESCENDING)
        .reduceGroup(new IdentityGroupReducerCombinable<Pojo4>())
        .output(new DiscardingOutputFormat<Pojo4>());
    Plan p = env.createProgramPlan();
    OptimizedPlan op = compileNoStats(p);
    SinkPlanNode sink = op.getDataSinks().iterator().next();
    SingleInputPlanNode reducer = (SingleInputPlanNode) sink.getInput().getSource();
    SingleInputPlanNode combiner = (SingleInputPlanNode) reducer.getInput().getSource();
    assertEquals(ShipStrategyType.FORWARD, sink.getInput().getShipStrategy());
    assertEquals(ShipStrategyType.PARTITION_CUSTOM, reducer.getInput().getShipStrategy());
    assertEquals(ShipStrategyType.FORWARD, combiner.getInput().getShipStrategy());
  }
  catch (Exception e) {
    e.printStackTrace();
    fail(e.getMessage());
  }
}

代码示例来源：origin: apache/flink

reduced.output(new DiscardingOutputFormat<Long>()).name("sink");

代码示例来源：origin: apache/flink

@Test
  public void testReduce() {
    // construct the plan
    ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
    env.setParallelism(DEFAULT_PARALLELISM);
    DataSet<Long> set1 = env.generateSequence(0,1);

    set1.reduceGroup(new IdentityGroupReducer<Long>()).name("Reduce1")
        .output(new DiscardingOutputFormat<Long>()).name("Sink");

    Plan plan = env.createProgramPlan();

    try {
      OptimizedPlan oPlan = compileNoStats(plan);
      JobGraphGenerator jobGen = new JobGraphGenerator();
      jobGen.compileJobGraph(oPlan);
    } catch(CompilerException ce) {
      ce.printStackTrace();
      fail("The pact compiler is unable to compile this plan correctly");
    }
  }
}

代码示例来源：origin: apache/flink

.reduceGroup(new IdentityGroupReducerCombinable<Tuple3<Long,Long,Long>>())
.output(new DiscardingOutputFormat<Tuple3<Long, Long, Long>>());

代码示例来源：origin: apache/flink

.output(new DiscardingOutputFormat<Tuple3<Long, Long, Long>>());

代码示例来源：origin: apache/flink

.reduceGroup(new IdentityGroupReducerCombinable<Tuple3<Long,Long,Long>>())
.output(new DiscardingOutputFormat<Tuple3<Long, Long, Long>>());

代码示例来源：origin: apache/flink

.output(new DiscardingOutputFormat<Long>());

代码示例来源：origin: apache/flink

.groupBy("*").reduceGroup(new IdentityGroupReducer<Long>())
  .withForwardedFields("*").setParallelism(p).name("Reduce2")
.output(new DiscardingOutputFormat<Long>()).setParallelism(p).name("Sink");

代码示例来源：origin: apache/flink

/**
   * Source -> Map -> Reduce -> Cross -> Reduce -> Cross -> Reduce ->
   * |--------------------------/                  /
   * |--------------------------------------------/
   * 
   * First cross has SameKeyFirst output contract
   */
  @Test
  public void testTicket158() {
    // construct the plan
    ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
    env.setParallelism(DEFAULT_PARALLELISM);
    DataSet<Long> set1 = env.generateSequence(0,1);

    set1.map(new IdentityMapper<Long>()).name("Map1")
        .groupBy("*").reduceGroup(new IdentityGroupReducer<Long>()).name("Reduce1")
        .cross(set1).with(new IdentityCrosser<Long>()).withForwardedFieldsFirst("*").name("Cross1")
        .groupBy("*").reduceGroup(new IdentityGroupReducer<Long>()).name("Reduce2")
        .cross(set1).with(new IdentityCrosser<Long>()).name("Cross2")
        .groupBy("*").reduceGroup(new IdentityGroupReducer<Long>()).name("Reduce3")
        .output(new DiscardingOutputFormat<Long>()).name("Sink");

    Plan plan = env.createProgramPlan();
    OptimizedPlan oPlan = compileNoStats(plan);

    JobGraphGenerator jobGen = new JobGraphGenerator();
    jobGen.compileJobGraph(oPlan);
  }
}

内容来源于网络，如有侵权，请联系作者删除！

GroupReduceOperator

相关文章

    查看更多
    微信公众号

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com