org.apache.flink.api.java.operators.GroupReduceOperator.withBroadcastSet()方法的使用及代码示例

x33g5p2x  于2022-01-20 转载在 其他  
字(4.6k)|赞(0)|评价(0)|浏览(106)

本文整理了Java中org.apache.flink.api.java.operators.GroupReduceOperator.withBroadcastSet()方法的一些代码示例,展示了GroupReduceOperator.withBroadcastSet()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。GroupReduceOperator.withBroadcastSet()方法的具体详情如下:
包路径:org.apache.flink.api.java.operators.GroupReduceOperator
类名称:GroupReduceOperator
方法名:withBroadcastSet

GroupReduceOperator.withBroadcastSet介绍

暂无

代码示例

代码示例来源:origin: apache/flink

@Override
protected void testProgram() throws Exception {
  ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
  DataSet<Integer> data = env.fromElements(1, 2, 3, 4, 5, 6, 7, 8);
  IterativeDataSet<Integer> iteration = data.iterate(10);
  DataSet<Integer> result = data.reduceGroup(new PickOneAllReduce()).withBroadcastSet(iteration, "bc");
  final List<Integer> resultList = new ArrayList<Integer>();
  iteration.closeWith(result).output(new LocalCollectionOutputFormat<Integer>(resultList));
  env.execute();
  Assert.assertEquals(8, resultList.get(0).intValue());
}

代码示例来源:origin: apache/flink

@Override
protected void testProgram() throws Exception {
  ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
  env.setParallelism(4);
  DataSet<Integer> data = env.fromElements(1, 2, 3, 4, 5, 6, 7, 8);
  IterativeDataSet<Integer> iteration = data.iterate(10);
  DataSet<Integer> result = data.reduceGroup(new PickOneAllReduce()).withBroadcastSet(iteration, "bc");
  final List<Integer> resultList = new ArrayList<Integer>();
  iteration.closeWith(result).output(new LocalCollectionOutputFormat<Integer>(resultList));
  env.execute();
  Assert.assertEquals(8, resultList.get(0).intValue());
}

代码示例来源:origin: apache/flink

@Test
public void testCorrectnessOfGroupReduceWithBroadcastSet() throws Exception {
  /*
   * check correctness of groupReduce with broadcast set
   */
  final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
  DataSet<Integer> intDs = CollectionDataSets.getIntegerDataSet(env);
  DataSet<Tuple3<Integer, Long, String>> ds = CollectionDataSets.get3TupleDataSet(env);
  DataSet<Tuple3<Integer, Long, String>> reduceDs = ds.
      groupBy(1).reduceGroup(new BCTuple3GroupReduce()).withBroadcastSet(intDs, "ints");
  List<Tuple3<Integer, Long, String>> result = reduceDs.collect();
  String expected = "1,1,55\n" +
      "5,2,55\n" +
      "15,3,55\n" +
      "34,4,55\n" +
      "65,5,55\n" +
      "111,6,55\n";
  compareResultAsTuples(result, expected);
}

代码示例来源:origin: apache/flink

iteration2.closeWith(iteration2.reduceGroup(new Top1GroupReducer<String>()).withBroadcastSet(reduced, "bc2"))
    .output(new DiscardingOutputFormat<String>());
iteration3.closeWith(iteration3.reduceGroup(new IdentityGroupReducer<String>()).withBroadcastSet(reduced, "bc3"))
    .output(new DiscardingOutputFormat<String>());

代码示例来源:origin: apache/flink

.map(new IdentityMapper<String>())
.reduceGroup(new Top1GroupReducer<String>())
  .withBroadcastSet(input3, "bc");
  .withBroadcastSet(input3, "bc");

代码示例来源:origin: apache/flink

.withBroadcastSet(sumOfScoresAndNumberOfNeighborPairs, SUM_OF_SCORES_AND_NUMBER_OF_NEIGHBOR_PAIRS);

代码示例来源:origin: org.gradoop/gradoop-flink

/**
 * Determines frequent subgraphs in a set of embeddings.
 *
 * @param embeddings set of embeddings
 * @return frequent subgraphs
 */
private DataSet<CCSSubgraph> getCategoryFrequentSubgraphs(
 DataSet<CCSSubgraphEmbeddings> embeddings) {
 return embeddings
  .map(new CCSSubgraphOnly())
  .groupBy(0, 3)
  .sum(1)
  .groupBy(0)
  .reduceGroup(new CategoryFrequentAndInteresting(minInterestingness))
  .withBroadcastSet(categoryCounts, TFSMConstants.GRAPH_COUNT)
  .withBroadcastSet(categoryMinFrequencies, DIMSpanConstants.MIN_FREQUENCY);
}

代码示例来源:origin: dbs-leipzig/gradoop

/**
 * Determines frequent subgraphs in a set of embeddings.
 *
 * @param embeddings set of embeddings
 * @return frequent subgraphs
 */
private DataSet<CCSSubgraph> getCategoryFrequentSubgraphs(
 DataSet<CCSSubgraphEmbeddings> embeddings) {
 return embeddings
  .map(new CCSSubgraphOnly())
  .groupBy(0, 3)
  .sum(1)
  .groupBy(0)
  .reduceGroup(new CategoryFrequentAndInteresting(minInterestingness))
  .withBroadcastSet(categoryCounts, TFSMConstants.GRAPH_COUNT)
  .withBroadcastSet(categoryMinFrequencies, DIMSpanConstants.MIN_FREQUENCY);
}

代码示例来源:origin: org.apache.flink/flink-gelly_2.10

.withBroadcastSet(sumOfScoresAndNumberOfNeighborPairs, SUM_OF_SCORES_AND_NUMBER_OF_NEIGHBOR_PAIRS);

代码示例来源:origin: com.alibaba.blink/flink-gelly

.withBroadcastSet(sumOfScoresAndNumberOfNeighborPairs, SUM_OF_SCORES_AND_NUMBER_OF_NEIGHBOR_PAIRS);

代码示例来源:origin: org.apache.flink/flink-gelly_2.11

.withBroadcastSet(sumOfScoresAndNumberOfNeighborPairs, SUM_OF_SCORES_AND_NUMBER_OF_NEIGHBOR_PAIRS);

相关文章