org.apache.spark.api.java.JavaRDD.flatMapToPair()方法的使用及代码示例

x33g5p2x  于2022-01-21 转载在 其他  
字(9.1k)|赞(0)|评价(0)|浏览(124)

本文整理了Java中org.apache.spark.api.java.JavaRDD.flatMapToPair()方法的一些代码示例，展示了JavaRDD.flatMapToPair()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。JavaRDD.flatMapToPair()方法的具体详情如下：
包路径：org.apache.spark.api.java.JavaRDD
类名称：JavaRDD
方法名：flatMapToPair

JavaRDD.flatMapToPair介绍

暂无

代码示例

代码示例来源：origin: OryxProject/oryx

public static Map<String,Integer> countDistinctOtherWords(JavaPairRDD<String,String> data) {
 return data.values().flatMapToPair(line -> {
  Set<String> distinctTokens = new HashSet<>(Arrays.asList(line.split(" ")));
  return distinctTokens.stream().flatMap(a ->
   distinctTokens.stream().filter(b -> !a.equals(b)).map(b -> new Tuple2<>(a, b))
  ).iterator();
 }).distinct().mapValues(a -> 1).reduceByKey((c1, c2) -> c1 + c2).collectAsMap();
}

代码示例来源：origin: OryxProject/oryx

examplesRDD.flatMapToPair(example -> {
 Feature target = example.getTarget();
 DecisionTree[] trees = forest.getTrees();

代码示例来源：origin: org.apache.spark/spark-core

@Test
public void flatMap() {
 JavaRDD<String> rdd = sc.parallelize(Arrays.asList("Hello World!",
  "The quick brown fox jumps over the lazy dog."));
 JavaRDD<String> words = rdd.flatMap(x -> Arrays.asList(x.split(" ")).iterator());
 Assert.assertEquals("Hello", words.first());
 Assert.assertEquals(11, words.count());
 JavaPairRDD<String, String> pairs = rdd.flatMapToPair(s -> {
  List<Tuple2<String, String>> pairs2 = new LinkedList<>();
  for (String word : s.split(" ")) {
   pairs2.add(new Tuple2<>(word, word));
  }
  return pairs2.iterator();
 });
 Assert.assertEquals(new Tuple2<>("Hello", "Hello"), pairs.first());
 Assert.assertEquals(11, pairs.count());
 JavaDoubleRDD doubles = rdd.flatMapToDouble(s -> {
  List<Double> lengths = new LinkedList<>();
  for (String word : s.split(" ")) {
   lengths.add((double) word.length());
  }
  return lengths.iterator();
 });
 Assert.assertEquals(5.0, doubles.first(), 0.01);
 Assert.assertEquals(11, pairs.count());
}

代码示例来源：origin: org.apache.spark/spark-core_2.10

@Test
public void flatMap() {
 JavaRDD<String> rdd = sc.parallelize(Arrays.asList("Hello World!",
  "The quick brown fox jumps over the lazy dog."));
 JavaRDD<String> words = rdd.flatMap(x -> Arrays.asList(x.split(" ")).iterator());
 Assert.assertEquals("Hello", words.first());
 Assert.assertEquals(11, words.count());
 JavaPairRDD<String, String> pairs = rdd.flatMapToPair(s -> {
  List<Tuple2<String, String>> pairs2 = new LinkedList<>();
  for (String word : s.split(" ")) {
   pairs2.add(new Tuple2<>(word, word));
  }
  return pairs2.iterator();
 });
 Assert.assertEquals(new Tuple2<>("Hello", "Hello"), pairs.first());
 Assert.assertEquals(11, pairs.count());
 JavaDoubleRDD doubles = rdd.flatMapToDouble(s -> {
  List<Double> lengths = new LinkedList<>();
  for (String word : s.split(" ")) {
   lengths.add((double) word.length());
  }
  return lengths.iterator();
 });
 Assert.assertEquals(5.0, doubles.first(), 0.01);
 Assert.assertEquals(11, pairs.count());
}

代码示例来源：origin: org.apache.spark/spark-core_2.11

@Test
public void flatMap() {
 JavaRDD<String> rdd = sc.parallelize(Arrays.asList("Hello World!",
  "The quick brown fox jumps over the lazy dog."));
 JavaRDD<String> words = rdd.flatMap(x -> Arrays.asList(x.split(" ")).iterator());
 assertEquals("Hello", words.first());
 assertEquals(11, words.count());
 JavaPairRDD<String, String> pairsRDD = rdd.flatMapToPair(s -> {
   List<Tuple2<String, String>> pairs = new LinkedList<>();
   for (String word : s.split(" ")) {
    pairs.add(new Tuple2<>(word, word));
   }
   return pairs.iterator();
  }
 );
 assertEquals(new Tuple2<>("Hello", "Hello"), pairsRDD.first());
 assertEquals(11, pairsRDD.count());
 JavaDoubleRDD doubles = rdd.flatMapToDouble(s -> {
  List<Double> lengths = new LinkedList<>();
  for (String word : s.split(" ")) {
   lengths.add((double) word.length());
  }
  return lengths.iterator();
 });
 assertEquals(5.0, doubles.first(), 0.01);
 assertEquals(11, pairsRDD.count());
}

代码示例来源：origin: org.apache.spark/spark-core_2.11

@Test
public void flatMap() {
 JavaRDD<String> rdd = sc.parallelize(Arrays.asList("Hello World!",
  "The quick brown fox jumps over the lazy dog."));
 JavaRDD<String> words = rdd.flatMap(x -> Arrays.asList(x.split(" ")).iterator());
 Assert.assertEquals("Hello", words.first());
 Assert.assertEquals(11, words.count());
 JavaPairRDD<String, String> pairs = rdd.flatMapToPair(s -> {
  List<Tuple2<String, String>> pairs2 = new LinkedList<>();
  for (String word : s.split(" ")) {
   pairs2.add(new Tuple2<>(word, word));
  }
  return pairs2.iterator();
 });
 Assert.assertEquals(new Tuple2<>("Hello", "Hello"), pairs.first());
 Assert.assertEquals(11, pairs.count());
 JavaDoubleRDD doubles = rdd.flatMapToDouble(s -> {
  List<Double> lengths = new LinkedList<>();
  for (String word : s.split(" ")) {
   lengths.add((double) word.length());
  }
  return lengths.iterator();
 });
 Assert.assertEquals(5.0, doubles.first(), 0.01);
 Assert.assertEquals(11, pairs.count());
}

代码示例来源：origin: org.apache.spark/spark-core_2.10

@Test
public void flatMap() {
 JavaRDD<String> rdd = sc.parallelize(Arrays.asList("Hello World!",
  "The quick brown fox jumps over the lazy dog."));
 JavaRDD<String> words = rdd.flatMap(x -> Arrays.asList(x.split(" ")).iterator());
 assertEquals("Hello", words.first());
 assertEquals(11, words.count());
 JavaPairRDD<String, String> pairsRDD = rdd.flatMapToPair(s -> {
   List<Tuple2<String, String>> pairs = new LinkedList<>();
   for (String word : s.split(" ")) {
    pairs.add(new Tuple2<>(word, word));
   }
   return pairs.iterator();
  }
 );
 assertEquals(new Tuple2<>("Hello", "Hello"), pairsRDD.first());
 assertEquals(11, pairsRDD.count());
 JavaDoubleRDD doubles = rdd.flatMapToDouble(s -> {
  List<Double> lengths = new LinkedList<>();
  for (String word : s.split(" ")) {
   lengths.add((double) word.length());
  }
  return lengths.iterator();
 });
 assertEquals(5.0, doubles.first(), 0.01);
 assertEquals(11, pairsRDD.count());
}

代码示例来源：origin: org.apache.spark/spark-core

@Test
public void flatMap() {
 JavaRDD<String> rdd = sc.parallelize(Arrays.asList("Hello World!",
  "The quick brown fox jumps over the lazy dog."));
 JavaRDD<String> words = rdd.flatMap(x -> Arrays.asList(x.split(" ")).iterator());
 assertEquals("Hello", words.first());
 assertEquals(11, words.count());
 JavaPairRDD<String, String> pairsRDD = rdd.flatMapToPair(s -> {
   List<Tuple2<String, String>> pairs = new LinkedList<>();
   for (String word : s.split(" ")) {
    pairs.add(new Tuple2<>(word, word));
   }
   return pairs.iterator();
  }
 );
 assertEquals(new Tuple2<>("Hello", "Hello"), pairsRDD.first());
 assertEquals(11, pairsRDD.count());
 JavaDoubleRDD doubles = rdd.flatMapToDouble(s -> {
  List<Double> lengths = new LinkedList<>();
  for (String word : s.split(" ")) {
   lengths.add((double) word.length());
  }
  return lengths.iterator();
 });
 assertEquals(5.0, doubles.first(), 0.01);
 assertEquals(11, pairsRDD.count());
}

代码示例来源：origin: mahmoudparsian/data-algorithms-book

JavaPairRDD<String, String> rdd = lines.flatMapToPair(
    (String line) -> Util.mapToKeyValueList(line, N).iterator()
);

代码示例来源：origin: mahmoudparsian/data-algorithms-book

JavaPairRDD<String,Integer> ones = records.flatMapToPair((String rec) -> {

代码示例来源：origin: mahmoudparsian/data-algorithms-book

JavaPairRDD<String, String> rdd = lines.flatMapToPair(
    (String line) -> Util.mapToKeyValueList(line, N).iterator()
);

代码示例来源：origin: mahmoudparsian/data-algorithms-book

JavaPairRDD<Tuple2<String,String>,Integer> pairs = training.flatMapToPair((String rec) -> {
  List<Tuple2<Tuple2<String,String>,Integer>> result =
      new ArrayList<Tuple2<Tuple2<String,String>,Integer>>();

代码示例来源：origin: mahmoudparsian/data-algorithms-book

JavaPairRDD<String, String> rdd = lines.flatMapToPair(
    (String line) -> Util.mapToKeyValueList(line, N).iterator()
);

代码示例来源：origin: mahmoudparsian/data-algorithms-book

JavaPairRDD<String, String> rdd = lines.flatMapToPair(
    new PairFlatMapFunction<String, String, String>() {
  @Override

代码示例来源：origin: mahmoudparsian/data-algorithms-book

JavaPairRDD<String, String> rdd = lines.flatMapToPair(
    new PairFlatMapFunction<String, String, String>() {
  @Override

代码示例来源：origin: mahmoudparsian/data-algorithms-book

JavaPairRDD<Tuple2<String,String>,Integer> pairs = training.flatMapToPair(new PairFlatMapFunction<

代码示例来源：origin: mahmoudparsian/data-algorithms-book

JavaPairRDD<Character,Long> chars = lines.flatMapToPair(new PairFlatMapFunction<String, Character, Long>() {
 @Override
 public Iterator<Tuple2<Character,Long>> call(String s) {

代码示例来源：origin: mahmoudparsian/data-algorithms-book

JavaPairRDD<String,Integer> ones = records.flatMapToPair(new PairFlatMapFunction

代码示例来源：origin: mahmoudparsian/data-algorithms-book

JavaPairRDD<Character,Long> chars = lines.flatMapToPair(new PairFlatMapFunction<String, Character, Long>() {
 @Override
 public Iterator<Tuple2<Character,Long>> call(String s) {

代码示例来源：origin: mahmoudparsian/data-algorithms-book

S.flatMapToPair(new PairFlatMapFunction<

内容来源于网络，如有侵权，请联系作者删除！

JavaRDD

相关文章

    查看更多
    微信公众号

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多

    JavaRDD类方法

    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com