org.apache.spark.api.java.JavaRDD.saveAsTextFile()方法的使用及代码示例

x33g5p2x  于2022-01-21 转载在 其他  
字(7.1k)|赞(0)|评价(0)|浏览(87)

本文整理了Java中org.apache.spark.api.java.JavaRDD.saveAsTextFile()方法的一些代码示例，展示了JavaRDD.saveAsTextFile()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。JavaRDD.saveAsTextFile()方法的具体详情如下：
包路径：org.apache.spark.api.java.JavaRDD
类名称：JavaRDD
方法名：saveAsTextFile

JavaRDD.saveAsTextFile介绍

暂无

代码示例

代码示例来源：origin: OryxProject/oryx

private static void saveFeaturesRDD(JavaPairRDD<Integer,float[]> features,
                  Path path,
                  Broadcast<? extends Map<Integer,String>> bIndexToID) {
 log.info("Saving features RDD to {}", path);
 features.map(keyAndVector -> {
  String id = bIndexToID.value().get(keyAndVector._1());
  float[] vector = keyAndVector._2();
  return TextUtils.joinJSON(Arrays.asList(id, vector));
 }).saveAsTextFile(path.toString(), GzipCodec.class);
}

代码示例来源：origin: databricks/learning-spark

public static void main(String[] args) throws Exception {
    if (args.length != 3) {
   throw new Exception("Usage BasicLoadCsv sparkMaster csvInputFile csvOutputFile key");
    }
  String master = args[0];
  String csvInput = args[1];
  String outputFile = args[2];
  final String key = args[3];

    JavaSparkContext sc = new JavaSparkContext(
   master, "loadwholecsv", System.getenv("SPARK_HOME"), System.getenv("JARS"));
  JavaPairRDD<String, String> csvData = sc.wholeTextFiles(csvInput);
  JavaRDD<String[]> keyedRDD = csvData.flatMap(new ParseLine());
  JavaRDD<String[]> result =
   keyedRDD.filter(new Function<String[], Boolean>() {
     public Boolean call(String[] input) { return input[0].equals(key); }});

  result.saveAsTextFile(outputFile);
  }
}

代码示例来源：origin: databricks/learning-spark

public static void main(String[] args) throws Exception {
    if (args.length != 3) {
   throw new Exception("Usage BasicLoadJson [sparkMaster] [jsoninput] [jsonoutput]");
    }
  String master = args[0];
  String fileName = args[1];
  String outfile = args[2];

    JavaSparkContext sc = new JavaSparkContext(
   master, "basicloadjson", System.getenv("SPARK_HOME"), System.getenv("JARS"));
  JavaRDD<String> input = sc.textFile(fileName);
  JavaRDD<Person> result = input.mapPartitions(new ParseJson()).filter(new LikesPandas());
  JavaRDD<String> formatted = result.mapPartitions(new WriteJson());
  formatted.saveAsTextFile(outfile);
  }
}

代码示例来源：origin: org.apache.spark/spark-core

@Test
public void textFiles() throws IOException {
 String outputDir = new File(tempDir, "output").getAbsolutePath();
 JavaRDD<Integer> rdd = sc.parallelize(Arrays.asList(1, 2, 3, 4));
 rdd.saveAsTextFile(outputDir);
 // Read the plain text file and check it's OK
 File outputFile = new File(outputDir, "part-00000");
 String content = Files.toString(outputFile, StandardCharsets.UTF_8);
 assertEquals("1\n2\n3\n4\n", content);
 // Also try reading it in as a text file RDD
 List<String> expected = Arrays.asList("1", "2", "3", "4");
 JavaRDD<String> readRDD = sc.textFile(outputDir);
 assertEquals(expected, readRDD.collect());
}

代码示例来源：origin: org.apache.spark/spark-core_2.11

@Test
public void textFilesCompressed() throws IOException {
 String outputDir = new File(tempDir, "output").getAbsolutePath();
 JavaRDD<Integer> rdd = sc.parallelize(Arrays.asList(1, 2, 3, 4));
 rdd.saveAsTextFile(outputDir, DefaultCodec.class);
 // Try reading it in as a text file RDD
 List<String> expected = Arrays.asList("1", "2", "3", "4");
 JavaRDD<String> readRDD = sc.textFile(outputDir);
 assertEquals(expected, readRDD.collect());
}

代码示例来源：origin: org.apache.spark/spark-core_2.10

@Test
public void textFilesCompressed() throws IOException {
 String outputDir = new File(tempDir, "output").getAbsolutePath();
 JavaRDD<Integer> rdd = sc.parallelize(Arrays.asList(1, 2, 3, 4));
 rdd.saveAsTextFile(outputDir, DefaultCodec.class);
 // Try reading it in as a text file RDD
 List<String> expected = Arrays.asList("1", "2", "3", "4");
 JavaRDD<String> readRDD = sc.textFile(outputDir);
 assertEquals(expected, readRDD.collect());
}

代码示例来源：origin: org.apache.spark/spark-core

@Test
public void textFilesCompressed() throws IOException {
 String outputDir = new File(tempDir, "output").getAbsolutePath();
 JavaRDD<Integer> rdd = sc.parallelize(Arrays.asList(1, 2, 3, 4));
 rdd.saveAsTextFile(outputDir, DefaultCodec.class);
 // Try reading it in as a text file RDD
 List<String> expected = Arrays.asList("1", "2", "3", "4");
 JavaRDD<String> readRDD = sc.textFile(outputDir);
 assertEquals(expected, readRDD.collect());
}

代码示例来源：origin: org.apache.spark/spark-core_2.10

@Test
public void textFiles() throws IOException {
 String outputDir = new File(tempDir, "output").getAbsolutePath();
 JavaRDD<Integer> rdd = sc.parallelize(Arrays.asList(1, 2, 3, 4));
 rdd.saveAsTextFile(outputDir);
 // Read the plain text file and check it's OK
 File outputFile = new File(outputDir, "part-00000");
 String content = Files.toString(outputFile, StandardCharsets.UTF_8);
 assertEquals("1\n2\n3\n4\n", content);
 // Also try reading it in as a text file RDD
 List<String> expected = Arrays.asList("1", "2", "3", "4");
 JavaRDD<String> readRDD = sc.textFile(outputDir);
 assertEquals(expected, readRDD.collect());
}

代码示例来源：origin: SeldonIO/seldon-server

json_only.saveAsTextFile(outputPath, org.apache.hadoop.io.compress.GzipCodec.class);
} else {
  json_only.saveAsTextFile(outputPath);

代码示例来源：origin: org.apache.spark/spark-core_2.11

@Test
public void textFiles() throws IOException {
 String outputDir = new File(tempDir, "output").getAbsolutePath();
 JavaRDD<Integer> rdd = sc.parallelize(Arrays.asList(1, 2, 3, 4));
 rdd.saveAsTextFile(outputDir);
 // Read the plain text file and check it's OK
 File outputFile = new File(outputDir, "part-00000");
 String content = Files.toString(outputFile, StandardCharsets.UTF_8);
 assertEquals("1\n2\n3\n4\n", content);
 // Also try reading it in as a text file RDD
 List<String> expected = Arrays.asList("1", "2", "3", "4");
 JavaRDD<String> readRDD = sc.textFile(outputDir);
 assertEquals(expected, readRDD.collect());
}

代码示例来源：origin: databricks/learning-spark

callSigns.saveAsTextFile(outputDir + "/callsigns");
System.out.println("Blank lines: "+ blankLines.value());

代码示例来源：origin: mahmoudparsian/data-algorithms-book

lines.saveAsTextFile("/output/1");
partitions.saveAsTextFile("/output/4");

代码示例来源：origin: mahmoudparsian/data-algorithms-book

lines.saveAsTextFile("/output/1");

代码示例来源：origin: mahmoudparsian/data-algorithms-book

lines.saveAsTextFile("/output/1");
partitions.saveAsTextFile("/output/4");

代码示例来源：origin: mahmoudparsian/data-algorithms-book

records.saveAsTextFile("/output/2");

代码示例来源：origin: mahmoudparsian/data-algorithms-book

lines.saveAsTextFile("/output/1");

代码示例来源：origin: mahmoudparsian/data-algorithms-book

biosets.saveAsTextFile("/ttest/output/1");

代码示例来源：origin: mahmoudparsian/data-algorithms-book

records.saveAsTextFile("/output/2");

代码示例来源：origin: mahmoudparsian/data-algorithms-book

biosets.saveAsTextFile("/ttest/output/1");

代码示例来源：origin: mahmoudparsian/data-algorithms-book

R.saveAsTextFile(outputPath+"/R");  
JavaRDD<String> S = session.read().textFile(datasetS).javaRDD();
S.saveAsTextFile(outputPath+"/S");

内容来源于网络，如有侵权，请联系作者删除！

JavaRDD

相关文章

    查看更多
    微信公众号

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多

    JavaRDD类方法

    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com