org.apache.spark.api.java.JavaRDD.wrapRDD()方法的使用及代码示例

x33g5p2x  于2022-01-21 转载在 其他  
字(2.9k)|赞(0)|评价(0)|浏览(97)

本文整理了Java中org.apache.spark.api.java.JavaRDD.wrapRDD()方法的一些代码示例,展示了JavaRDD.wrapRDD()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。JavaRDD.wrapRDD()方法的具体详情如下:
包路径:org.apache.spark.api.java.JavaRDD
类名称:JavaRDD
方法名:wrapRDD

JavaRDD.wrapRDD介绍

暂无

代码示例

代码示例来源:origin: OryxProject/oryx

/**
 * Default implementation which randomly splits new data into train/test sets.
 * This handles the case where {@link #getTestFraction()} is not 0 or 1.
 *
 * @param newData data that has arrived in the current input batch
 * @return a {@link Pair} of train, test {@link RDD}s.
 */
protected Pair<JavaRDD<M>,JavaRDD<M>> splitNewDataToTrainTest(JavaRDD<M> newData) {
 RDD<M>[] testTrainRDDs = newData.rdd().randomSplit(
   new double[]{1.0 - testFraction, testFraction},
   RandomManager.getRandom().nextLong());
 return new Pair<>(newData.wrapRDD(testTrainRDDs[0]),
          newData.wrapRDD(testTrainRDDs[1]));
}

代码示例来源:origin: OryxProject/oryx

private static JavaPairRDD<Integer,Iterable<Rating>> predictAll(
  MatrixFactorizationModel mfModel,
  JavaRDD<Rating> data,
  JavaPairRDD<Integer,Integer> userProducts) {
 @SuppressWarnings("unchecked")
 RDD<Tuple2<Object,Object>> userProductsRDD =
   (RDD<Tuple2<Object,Object>>) (RDD<?>) userProducts.rdd();
 return data.wrapRDD(mfModel.predict(userProductsRDD)).groupBy(Rating::user);
}

代码示例来源:origin: OryxProject/oryx

/**
 * Computes root mean squared error of {@link Rating#rating()} versus predicted value.
 */
static double rmse(MatrixFactorizationModel mfModel, JavaRDD<Rating> testData) {
 JavaPairRDD<Tuple2<Integer,Integer>,Double> testUserProductValues =
   testData.mapToPair(rating -> new Tuple2<>(new Tuple2<>(rating.user(), rating.product()), rating.rating()));
 @SuppressWarnings("unchecked")
 RDD<Tuple2<Object,Object>> testUserProducts =
   (RDD<Tuple2<Object,Object>>) (RDD<?>) testUserProductValues.keys().rdd();
 JavaRDD<Rating> predictions = testData.wrapRDD(mfModel.predict(testUserProducts));
 double mse = predictions.mapToPair(
   rating -> new Tuple2<>(new Tuple2<>(rating.user(), rating.product()), rating.rating())
 ).join(testUserProductValues).values().mapToDouble(valuePrediction -> {
  double diff = valuePrediction._1() - valuePrediction._2();
  return diff * diff;
 }).mean();
 return Math.sqrt(mse);
}

代码示例来源:origin: com.cloudera.oryx/oryx-ml

/**
 * Default implementation which randomly splits new data into train/test sets.
 * This handles the case where {@link #getTestFraction()} is not 0 or 1.
 *
 * @param newData data that has arrived in the current input batch
 * @return a {@link Pair} of train, test {@link RDD}s.
 */
protected Pair<JavaRDD<M>,JavaRDD<M>> splitNewDataToTrainTest(JavaRDD<M> newData) {
 RDD<M>[] testTrainRDDs = newData.rdd().randomSplit(
   new double[]{1.0 - testFraction, testFraction},
   RandomManager.getRandom().nextLong());
 return new Pair<>(newData.wrapRDD(testTrainRDDs[0]),
          newData.wrapRDD(testTrainRDDs[1]));
}

相关文章

微信公众号

最新文章

更多