org.apache.flink.api.java.operators.GroupReduceOperator.withParameters()方法的使用及代码示例

x33g5p2x  于2022-01-20 转载在 其他  
字(3.5k)|赞(0)|评价(0)|浏览(95)

本文整理了Java中org.apache.flink.api.java.operators.GroupReduceOperator.withParameters()方法的一些代码示例,展示了GroupReduceOperator.withParameters()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。GroupReduceOperator.withParameters()方法的具体详情如下:
包路径:org.apache.flink.api.java.operators.GroupReduceOperator
类名称:GroupReduceOperator
方法名:withParameters

GroupReduceOperator.withParameters介绍

暂无

代码示例

代码示例来源:origin: apache/flink

@Test
public void testCorrectnessOfAllGroupReduceForTuplesWithCombine() throws Exception {
  /*
   * check correctness of all-groupreduce for tuples with combine
   */
  org.junit.Assume.assumeTrue(mode != TestExecutionMode.COLLECTION);
  final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
  DataSet<Tuple3<Integer, Long, String>> ds = CollectionDataSets.get3TupleDataSet(env)
      .map(new IdentityMapper<Tuple3<Integer, Long, String>>()).setParallelism(4);
  Configuration cfg = new Configuration();
  cfg.setString(Optimizer.HINT_SHIP_STRATEGY, Optimizer.HINT_SHIP_STRATEGY_REPARTITION);
  DataSet<Tuple2<Integer, String>> reduceDs = ds.reduceGroup(new Tuple3AllGroupReduceWithCombine())
      .withParameters(cfg);
  List<Tuple2<Integer, String>> result = reduceDs.collect();
  String expected = "322," +
      "testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest\n";
  compareResultAsTuples(result, expected);
}

代码示例来源:origin: dataArtisans/cascading-flink

.reduceGroup(new GroupByReducer(node))
.returns(new TupleTypeInfo(outFields))
.withParameters(this.getFlinkNodeConfig(node))
.setParallelism(dop)
.name("reduce-" + node.getID());
.reduceGroup(new GroupByReducer(node))
.returns(new TupleTypeInfo(outFields))
.withParameters(this.getFlinkNodeConfig(node))
.setParallelism(dop)
.name("reduce-" + node.getID());

代码示例来源:origin: dataArtisans/cascading-flink

private DataSet<Tuple> translateGlobalGroupBy(DataSet<Tuple> input, FlowNode node, int dop,
                        String[] sortKeys, Order sortOrder, Fields outFields) {
  DataSet<Tuple> result = input;
  // sort on sorting keys if necessary
  if(sortKeys != null && sortKeys.length > 0) {
    result = result
        .sortPartition(sortKeys[0], sortOrder)
        .setParallelism(1)
        .name("reduce-"+ node.getID());
    for(int i=1; i<sortKeys.length; i++) {
      result = result
          .sortPartition(sortKeys[i], sortOrder)
          .setParallelism(1);
    }
  }
  // group all data
  return result
      .reduceGroup(new GroupByReducer(node))
      .returns(new TupleTypeInfo(outFields))
      .withParameters(this.getFlinkNodeConfig(node))
      .setParallelism(dop)
      .name("reduce-"+ node.getID());
}

代码示例来源:origin: dataArtisans/cascading-flink

.groupBy(groupingKeys)
    .reduceGroup(new CoGroupReducer(node))
    .withParameters(this.getFlinkNodeConfig(node))
    .setParallelism(dop)
    .returns(new TupleTypeInfo(outFields))
DataSet<Tuple> joinResult = ((DataSet<Tuple2<Tuple, Tuple[]>>) input)
    .reduceGroup(new CoGroupReducer(node))
    .withParameters(this.getFlinkNodeConfig(node))
    .setParallelism(1)
    .returns(new TupleTypeInfo(outFields))
    .sortGroup(1, Order.DESCENDING)
    .reduceGroup(new CoGroupBufferReducer(node))
    .withParameters(this.getFlinkNodeConfig(node))
    .setParallelism(dop)
    .returns(new TupleTypeInfo(outFields))
    .setParallelism(1)
    .reduceGroup(new CoGroupBufferReducer(node))
    .withParameters(this.getFlinkNodeConfig(node))
    .setParallelism(1)
    .returns(new TupleTypeInfo(outFields))

代码示例来源:origin: dataArtisans/cascading-flink

.reduceGroup(new GroupByReducer(node))
.returns(new TupleTypeInfo(outFields))
.withParameters(this.getFlinkNodeConfig(node))
.setParallelism(dop)
.name("reduce-" + node.getID());

相关文章