org.apache.spark.api.java.JavaPairRDD.isEmpty()方法的使用及代码示例

x33g5p2x  于2022-01-21 转载在 其他  
字(3.7k)|赞(0)|评价(0)|浏览(83)

本文整理了Java中org.apache.spark.api.java.JavaPairRDD.isEmpty()方法的一些代码示例,展示了JavaPairRDD.isEmpty()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。JavaPairRDD.isEmpty()方法的具体详情如下:
包路径:org.apache.spark.api.java.JavaPairRDD
类名称:JavaPairRDD
方法名:isEmpty

JavaPairRDD.isEmpty介绍

暂无

代码示例

代码示例来源:origin: OryxProject/oryx

@Override
public void call(JavaPairRDD<K,M> newData) throws IOException {
 if (newData.isEmpty()) {
  log.debug("RDD was empty");
 } else {
  Iterable<U> updates = modelManager.buildUpdates(newData);
  if (updates != null) {
   try (TopicProducer<String, U> producer = new TopicProducerImpl<>(updateBroker, updateTopic, true)) {
    updates.forEach(update -> producer.send("UP", update));
   }
  }
 }
}

代码示例来源:origin: OryxProject/oryx

@Override
 public void call(JavaPairRDD<K,M> rdd, Time time) throws IOException {
  if (rdd.isEmpty()) {
   log.info("RDD was empty, not saving to HDFS");
  } else {
   String file = prefix + "-" + time.milliseconds() + "." + suffix;
   Path path = new Path(file);
   FileSystem fs = FileSystem.get(path.toUri(), hadoopConf);
   if (fs.exists(path)) {
    log.warn("Saved data already existed, possibly from a failed job. Deleting {}", path);
    fs.delete(path, true);
   }
   log.info("Saving RDD to HDFS at {}", file);
   rdd.mapToPair(
     new ValueToWritableFunction<>(keyClass, messageClass, keyWritableClass, messageWritableClass)
   ).saveAsNewAPIHadoopFile(
     file,
     keyWritableClass,
     messageWritableClass,
     SequenceFileOutputFormat.class,
     hadoopConf);
  }
 }
}

代码示例来源:origin: OryxProject/oryx

throws IOException, InterruptedException {
if (newData.isEmpty()) {
 log.info("No data in current generation's RDD; nothing to do");
 return;

代码示例来源:origin: com.davidbracewell/mango

@Override
public boolean isEmpty() {
 return rdd.isEmpty();
}

代码示例来源:origin: com.cloudera.oryx/oryx-lambda

@Override
public void call(JavaPairRDD<K,M> newData) throws IOException {
 if (newData.isEmpty()) {
  log.debug("RDD was empty");
 } else {
  Iterable<U> updates = modelManager.buildUpdates(newData);
  if (updates != null) {
   try (TopicProducer<String, U> producer = new TopicProducerImpl<>(updateBroker, updateTopic, true)) {
    updates.forEach(update -> producer.send("UP", update));
   }
  }
 }
}

代码示例来源:origin: Stratio/Decision

@Override
public Void call(JavaPairRDD<StreamAction, Iterable<StratioStreamingMessage>> rdd) throws Exception {
  if (!rdd.isEmpty()) {
    rdd.mapPartitions(
        new FlatMapFunction<Iterator<Tuple2<StreamAction, Iterable<StratioStreamingMessage>>>, Object>() {
          @Override public Iterable<Object> call(
              Iterator<Tuple2<StreamAction, Iterable<StratioStreamingMessage>>> tuple2Iterator)
              throws Exception {
            while (tuple2Iterator.hasNext()) {
              process(tuple2Iterator.next()._2());
            }
            return new ArrayList<Object>();
          }
        }).count();
  }
  return null;
}

代码示例来源:origin: com.cloudera.oryx/oryx-lambda

@Override
 public void call(JavaPairRDD<K,M> rdd, Time time) throws IOException {
  if (rdd.isEmpty()) {
   log.info("RDD was empty, not saving to HDFS");
  } else {
   String file = prefix + "-" + time.milliseconds() + "." + suffix;
   Path path = new Path(file);
   FileSystem fs = FileSystem.get(path.toUri(), hadoopConf);
   if (fs.exists(path)) {
    log.warn("Saved data already existed, possibly from a failed job. Deleting {}", path);
    fs.delete(path, true);
   }
   log.info("Saving RDD to HDFS at {}", file);
   rdd.mapToPair(
     new ValueToWritableFunction<>(keyClass, messageClass, keyWritableClass, messageWritableClass)
   ).saveAsNewAPIHadoopFile(
     file,
     keyWritableClass,
     messageWritableClass,
     SequenceFileOutputFormat.class,
     hadoopConf);
  }
 }
}

代码示例来源:origin: com.cloudera.oryx/oryx-lambda

throws IOException, InterruptedException {
if (newData.isEmpty()) {
 log.info("No data in current generation's RDD; nothing to do");
 return;

代码示例来源:origin: locationtech/geowave

combinedResults.isEmpty();

相关文章

微信公众号

最新文章

更多

JavaPairRDD类方法