本文整理了Java中org.apache.flink.api.java.operators.GroupReduceOperator.withBroadcastSet()
方法的一些代码示例,展示了GroupReduceOperator.withBroadcastSet()
的具体用法。这些代码示例主要来源于Github
/Stackoverflow
/Maven
等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。GroupReduceOperator.withBroadcastSet()
方法的具体详情如下:
包路径:org.apache.flink.api.java.operators.GroupReduceOperator
类名称:GroupReduceOperator
方法名:withBroadcastSet
暂无
代码示例来源:origin: apache/flink
@Override
protected void testProgram() throws Exception {
ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
DataSet<Integer> data = env.fromElements(1, 2, 3, 4, 5, 6, 7, 8);
IterativeDataSet<Integer> iteration = data.iterate(10);
DataSet<Integer> result = data.reduceGroup(new PickOneAllReduce()).withBroadcastSet(iteration, "bc");
final List<Integer> resultList = new ArrayList<Integer>();
iteration.closeWith(result).output(new LocalCollectionOutputFormat<Integer>(resultList));
env.execute();
Assert.assertEquals(8, resultList.get(0).intValue());
}
代码示例来源:origin: apache/flink
@Override
protected void testProgram() throws Exception {
ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
env.setParallelism(4);
DataSet<Integer> data = env.fromElements(1, 2, 3, 4, 5, 6, 7, 8);
IterativeDataSet<Integer> iteration = data.iterate(10);
DataSet<Integer> result = data.reduceGroup(new PickOneAllReduce()).withBroadcastSet(iteration, "bc");
final List<Integer> resultList = new ArrayList<Integer>();
iteration.closeWith(result).output(new LocalCollectionOutputFormat<Integer>(resultList));
env.execute();
Assert.assertEquals(8, resultList.get(0).intValue());
}
代码示例来源:origin: apache/flink
@Test
public void testCorrectnessOfGroupReduceWithBroadcastSet() throws Exception {
/*
* check correctness of groupReduce with broadcast set
*/
final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
DataSet<Integer> intDs = CollectionDataSets.getIntegerDataSet(env);
DataSet<Tuple3<Integer, Long, String>> ds = CollectionDataSets.get3TupleDataSet(env);
DataSet<Tuple3<Integer, Long, String>> reduceDs = ds.
groupBy(1).reduceGroup(new BCTuple3GroupReduce()).withBroadcastSet(intDs, "ints");
List<Tuple3<Integer, Long, String>> result = reduceDs.collect();
String expected = "1,1,55\n" +
"5,2,55\n" +
"15,3,55\n" +
"34,4,55\n" +
"65,5,55\n" +
"111,6,55\n";
compareResultAsTuples(result, expected);
}
代码示例来源:origin: apache/flink
iteration2.closeWith(iteration2.reduceGroup(new Top1GroupReducer<String>()).withBroadcastSet(reduced, "bc2"))
.output(new DiscardingOutputFormat<String>());
iteration3.closeWith(iteration3.reduceGroup(new IdentityGroupReducer<String>()).withBroadcastSet(reduced, "bc3"))
.output(new DiscardingOutputFormat<String>());
代码示例来源:origin: apache/flink
.map(new IdentityMapper<String>())
.reduceGroup(new Top1GroupReducer<String>())
.withBroadcastSet(input3, "bc");
.withBroadcastSet(input3, "bc");
代码示例来源:origin: apache/flink
.withBroadcastSet(sumOfScoresAndNumberOfNeighborPairs, SUM_OF_SCORES_AND_NUMBER_OF_NEIGHBOR_PAIRS);
代码示例来源:origin: org.gradoop/gradoop-flink
/**
* Determines frequent subgraphs in a set of embeddings.
*
* @param embeddings set of embeddings
* @return frequent subgraphs
*/
private DataSet<CCSSubgraph> getCategoryFrequentSubgraphs(
DataSet<CCSSubgraphEmbeddings> embeddings) {
return embeddings
.map(new CCSSubgraphOnly())
.groupBy(0, 3)
.sum(1)
.groupBy(0)
.reduceGroup(new CategoryFrequentAndInteresting(minInterestingness))
.withBroadcastSet(categoryCounts, TFSMConstants.GRAPH_COUNT)
.withBroadcastSet(categoryMinFrequencies, DIMSpanConstants.MIN_FREQUENCY);
}
代码示例来源:origin: dbs-leipzig/gradoop
/**
* Determines frequent subgraphs in a set of embeddings.
*
* @param embeddings set of embeddings
* @return frequent subgraphs
*/
private DataSet<CCSSubgraph> getCategoryFrequentSubgraphs(
DataSet<CCSSubgraphEmbeddings> embeddings) {
return embeddings
.map(new CCSSubgraphOnly())
.groupBy(0, 3)
.sum(1)
.groupBy(0)
.reduceGroup(new CategoryFrequentAndInteresting(minInterestingness))
.withBroadcastSet(categoryCounts, TFSMConstants.GRAPH_COUNT)
.withBroadcastSet(categoryMinFrequencies, DIMSpanConstants.MIN_FREQUENCY);
}
代码示例来源:origin: org.apache.flink/flink-gelly_2.10
.withBroadcastSet(sumOfScoresAndNumberOfNeighborPairs, SUM_OF_SCORES_AND_NUMBER_OF_NEIGHBOR_PAIRS);
代码示例来源:origin: com.alibaba.blink/flink-gelly
.withBroadcastSet(sumOfScoresAndNumberOfNeighborPairs, SUM_OF_SCORES_AND_NUMBER_OF_NEIGHBOR_PAIRS);
代码示例来源:origin: org.apache.flink/flink-gelly_2.11
.withBroadcastSet(sumOfScoresAndNumberOfNeighborPairs, SUM_OF_SCORES_AND_NUMBER_OF_NEIGHBOR_PAIRS);
内容来源于网络,如有侵权,请联系作者删除!