org.apache.spark.api.java.JavaRDD.fromRDD()方法的使用及代码示例

x33g5p2x  于2022-01-21 转载在 其他  
字(1.7k)|赞(0)|评价(0)|浏览(95)

本文整理了Java中org.apache.spark.api.java.JavaRDD.fromRDD()方法的一些代码示例,展示了JavaRDD.fromRDD()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。JavaRDD.fromRDD()方法的具体详情如下:
包路径:org.apache.spark.api.java.JavaRDD
类名称:JavaRDD
方法名:fromRDD

JavaRDD.fromRDD介绍

暂无

代码示例

代码示例来源:origin: spirom/learning-spark-with-java

JavaRDD.fromRDD(combinedRDD.rdd(), combinedRDD.classTag());
JavaRDD<Tuple2<String, Double>> tupleDividedRDD = tupleCombinedRDD.map(keyAndsumAndCount ->
  new Tuple2<>(keyAndsumAndCount._1(), (double) keyAndsumAndCount._2()._1() / keyAndsumAndCount._2()._2()));

代码示例来源:origin: sujee81/SparkApps

public static void main(String[] args) {
  DbConnection dbConnection = new DbConnection(MYSQL_DRIVER, MYSQL_CONNECTION_URL, MYSQL_USERNAME, MYSQL_PWD);
  // Load data from MySQL
  JdbcRDD<Object[]> jdbcRDD =
      new JdbcRDD<>(sc.sc(), dbConnection, "select * from employees where emp_no >= ? and emp_no <= ?", 10001,
             499999, 10, new MapResult(), ClassManifestFactory$.MODULE$.fromClass(Object[].class));
  // Convert to JavaRDD
  JavaRDD<Object[]> javaRDD = JavaRDD.fromRDD(jdbcRDD, ClassManifestFactory$.MODULE$.fromClass(Object[].class));
  // Join first name and last name
  List<String> employeeFullNameList = javaRDD.map(new Function<Object[], String>() {
    @Override
    public String call(final Object[] record) throws Exception {
      return record[2] + " " + record[3];
    }
  }).collect();
  for (String fullName : employeeFullNameList) {
    LOGGER.info(fullName);
  }
}

代码示例来源:origin: org.apache.pig/pig

} else {
  for (RDD<Tuple> rdd : predecessors) {
    JavaRDD<Tuple> jrdd = JavaRDD.fromRDD(rdd, SparkUtil.getManifest(Tuple.class));
    JavaRDD<Tuple2<IndexedKey, Tuple>> rddPair = jrdd.map(new ToKeyValueFunction());
    rddPairs.add(rddPair.rdd());

相关文章

微信公众号

最新文章

更多