org.apache.spark.api.java.JavaPairRDD.partitioner()方法的使用及代码示例

x33g5p2x  于2022-01-21 转载在 其他  
字(9.1k)|赞(0)|评价(0)|浏览(77)

本文整理了Java中org.apache.spark.api.java.JavaPairRDD.partitioner()方法的一些代码示例，展示了JavaPairRDD.partitioner()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。JavaPairRDD.partitioner()方法的具体详情如下：
包路径：org.apache.spark.api.java.JavaPairRDD
类名称：JavaPairRDD
方法名：partitioner

JavaPairRDD.partitioner介绍

暂无

代码示例

代码示例来源：origin: org.apache.spark/spark-core

assertTrue(repartitioned.partitioner().isPresent());
assertEquals(repartitioned.partitioner().get(), partitioner);
List<List<Tuple2<Integer, Integer>>> partitions = repartitioned.glom().collect();
assertEquals(partitions.get(0),

代码示例来源：origin: org.apache.spark/spark-core_2.11

assertTrue(repartitioned.partitioner().isPresent());
assertEquals(repartitioned.partitioner().get(), partitioner);
List<List<Tuple2<Integer, Integer>>> partitions = repartitioned.glom().collect();
assertEquals(partitions.get(0),

代码示例来源：origin: org.apache.spark/spark-core_2.10

assertTrue(repartitioned.partitioner().isPresent());
assertEquals(repartitioned.partitioner().get(), partitioner);
List<List<Tuple2<Integer, Integer>>> partitions = repartitioned.glom().collect();
assertEquals(partitions.get(0),

代码示例来源：origin: apache/tinkerpop

public static <M> JavaPairRDD<Object, VertexWritable> prepareFinalGraphRDD(
    final JavaPairRDD<Object, VertexWritable> graphRDD,
    final JavaPairRDD<Object, ViewIncomingPayload<M>> viewIncomingRDD,
    final Set<VertexComputeKey> vertexComputeKeys) {
  // the graphRDD and the viewRDD must have the same partitioner
  if (graphRDD.partitioner().isPresent())
    assert (graphRDD.partitioner().get().equals(viewIncomingRDD.partitioner().get()));
  final String[] vertexComputeKeysArray = VertexProgramHelper.vertexComputeKeysAsArray(vertexComputeKeys); // the compute keys as an array
  return graphRDD.leftOuterJoin(viewIncomingRDD)
      .mapValues(tuple -> {
        final StarGraph.StarVertex vertex = tuple._1().get();
        vertex.dropVertexProperties(vertexComputeKeysArray); // drop all existing compute keys
        // attach the final computed view to the cached graph
        final List<DetachedVertexProperty<Object>> view = tuple._2().isPresent() ? tuple._2().get().getView() : Collections.emptyList();
        for (final DetachedVertexProperty<Object> property : view) {
          if (!VertexProgramHelper.isTransientVertexComputeKey(property.key(), vertexComputeKeys))
            property.attach(Attachable.Method.create(vertex));
        }
        return tuple._1();
      });
}

代码示例来源：origin: apache/tinkerpop

if (loadedGraphRDD.partitioner().isPresent())
  this.logger.debug("Using the existing partitioner associated with the loaded graphRDD: " + loadedGraphRDD.partitioner().get());
else {
  if (!skipPartitioner) {
    loadedGraphRDD = loadedGraphRDD.partitionBy(partitioner);
    partitioned = true;
    assert loadedGraphRDD.partitioner().isPresent();
  } else {
    assert skipPartitioner == !loadedGraphRDD.partitioner().isPresent(); // no easy way to test this with a test case
    this.logger.debug("Partitioning has been skipped for the loaded graphRDD via " + GREMLIN_SPARK_SKIP_PARTITIONER);

代码示例来源：origin: apache/tinkerpop

final Configuration vertexProgramConfiguration) { // has the VertexProgram.loadState() information
boolean partitionedGraphRDD = graphRDD.partitioner().isPresent();
  assert graphRDD.partitioner().get().equals(viewIncomingRDD.partitioner().get());
final JavaPairRDD<Object, ViewOutgoingPayload<M>> viewOutgoingRDD = ((null == viewIncomingRDD) ?
    graphRDD.mapValues(vertexWritable -> new Tuple2<>(vertexWritable, Optional.<ViewIncomingPayload<M>>absent())) : // first iteration will not have any views or messages
  assert graphRDD.partitioner().get().equals(viewOutgoingRDD.partitioner().get());
        viewOutgoingRDD.flatMapToPair(messageFunction).reduceByKey(graphRDD.partitioner().get(), reducerFunction) :
        viewOutgoingRDD.flatMapToPair(messageFunction).reduceByKey(reducerFunction))
  assert graphRDD.partitioner().get().equals(newViewIncomingRDD.partitioner().get());
newViewIncomingRDD
    .foreachPartition(partitionIterator -> {

代码示例来源：origin: ai.grakn/grakn-kb

public static <M> JavaPairRDD<Object, VertexWritable> prepareFinalGraphRDD(
    final JavaPairRDD<Object, VertexWritable> graphRDD,
    final JavaPairRDD<Object, ViewIncomingPayload<M>> viewIncomingRDD,
    final Set<VertexComputeKey> vertexComputeKeys) {
  // the graphRDD and the viewRDD must have the same partitioner
  if (graphRDD.partitioner().isPresent()){
    assert (graphRDD.partitioner().get().equals(viewIncomingRDD.partitioner().get()));}
  final String[] vertexComputeKeysArray = VertexProgramHelper.vertexComputeKeysAsArray(vertexComputeKeys); // the compute keys as an array
  return graphRDD.leftOuterJoin(viewIncomingRDD)
      .mapValues(tuple -> {
        final StarGraph.StarVertex vertex = tuple._1().get();
        vertex.dropVertexProperties(vertexComputeKeysArray); // drop all existing compute keys
        // attach the final computed view to the cached graph
        final List<DetachedVertexProperty<Object>> view = tuple._2().isPresent() ? tuple._2().get().getView() : Collections.emptyList();
        for (final DetachedVertexProperty<Object> property : view) {
          if (!VertexProgramHelper.isTransientVertexComputeKey(property.key(), vertexComputeKeys)){
            property.attach(Attachable.Method.create(vertex));}
        }
        return tuple._1();
      });
}

代码示例来源：origin: org.apache.tinkerpop/spark-gremlin

public static <M> JavaPairRDD<Object, VertexWritable> prepareFinalGraphRDD(
    final JavaPairRDD<Object, VertexWritable> graphRDD,
    final JavaPairRDD<Object, ViewIncomingPayload<M>> viewIncomingRDD,
    final Set<VertexComputeKey> vertexComputeKeys) {
  // the graphRDD and the viewRDD must have the same partitioner
  if (graphRDD.partitioner().isPresent())
    assert (graphRDD.partitioner().get().equals(viewIncomingRDD.partitioner().get()));
  final String[] vertexComputeKeysArray = VertexProgramHelper.vertexComputeKeysAsArray(vertexComputeKeys); // the compute keys as an array
  return graphRDD.leftOuterJoin(viewIncomingRDD)
      .mapValues(tuple -> {
        final StarGraph.StarVertex vertex = tuple._1().get();
        vertex.dropVertexProperties(vertexComputeKeysArray); // drop all existing compute keys
        // attach the final computed view to the cached graph
        final List<DetachedVertexProperty<Object>> view = tuple._2().isPresent() ? tuple._2().get().getView() : Collections.emptyList();
        for (final DetachedVertexProperty<Object> property : view) {
          if (!VertexProgramHelper.isTransientVertexComputeKey(property.key(), vertexComputeKeys))
            property.attach(Attachable.Method.create(vertex));
        }
        return tuple._1();
      });
}

代码示例来源：origin: org.apache.tinkerpop/spark-gremlin

if (loadedGraphRDD.partitioner().isPresent())
  this.logger.debug("Using the existing partitioner associated with the loaded graphRDD: " + loadedGraphRDD.partitioner().get());
else {
  if (!skipPartitioner) {
    loadedGraphRDD = loadedGraphRDD.partitionBy(partitioner);
    partitioned = true;
    assert loadedGraphRDD.partitioner().isPresent();
  } else {
    assert skipPartitioner == !loadedGraphRDD.partitioner().isPresent(); // no easy way to test this with a test case
    this.logger.debug("Partitioning has been skipped for the loaded graphRDD via " + GREMLIN_SPARK_SKIP_PARTITIONER);

代码示例来源：origin: ai.grakn/grakn-kb

if (loadedGraphRDD.partitioner().isPresent()) {
  this.logger.debug("Using the existing partitioner associated with the loaded graphRDD: " +
      loadedGraphRDD.partitioner().get());
} else {
  if (!skipPartitioner) {
    loadedGraphRDD = loadedGraphRDD.partitionBy(partitioner);
    partitioned = true;
    assert loadedGraphRDD.partitioner().isPresent();
  } else {
    assert skipPartitioner == !loadedGraphRDD.partitioner().isPresent();

代码示例来源：origin: ai.grakn/grakn-kb

final Configuration vertexProgramConfiguration) { // has the VertexProgram.loadState() information
boolean partitionedGraphRDD = graphRDD.partitioner().isPresent();
{assert graphRDD.partitioner().get().equals(viewIncomingRDD.partitioner().get());}
final JavaPairRDD<Object, ViewOutgoingPayload<M>> viewOutgoingRDD = ((null == viewIncomingRDD) ?
    graphRDD.mapValues(vertexWritable -> new Tuple2<>(vertexWritable, Optional.<ViewIncomingPayload<M>>absent())) : // first iteration will not have any views or messages
  assert graphRDD.partitioner().get().equals(viewOutgoingRDD.partitioner().get());}
        viewOutgoingRDD.flatMapToPair(messageFunction).reduceByKey(graphRDD.partitioner().get(), reducerFunction) :
        viewOutgoingRDD.flatMapToPair(messageFunction).reduceByKey(reducerFunction))
  assert graphRDD.partitioner().get().equals(newViewIncomingRDD.partitioner().get());}
newViewIncomingRDD
    .foreachPartition(partitionIterator -> {

代码示例来源：origin: org.apache.tinkerpop/spark-gremlin

final Configuration vertexProgramConfiguration) { // has the VertexProgram.loadState() information
boolean partitionedGraphRDD = graphRDD.partitioner().isPresent();
  assert graphRDD.partitioner().get().equals(viewIncomingRDD.partitioner().get());
final JavaPairRDD<Object, ViewOutgoingPayload<M>> viewOutgoingRDD = ((null == viewIncomingRDD) ?
    graphRDD.mapValues(vertexWritable -> new Tuple2<>(vertexWritable, Optional.<ViewIncomingPayload<M>>absent())) : // first iteration will not have any views or messages
  assert graphRDD.partitioner().get().equals(viewOutgoingRDD.partitioner().get());
        viewOutgoingRDD.flatMapToPair(messageFunction).reduceByKey(graphRDD.partitioner().get(), reducerFunction) :
        viewOutgoingRDD.flatMapToPair(messageFunction).reduceByKey(reducerFunction))
  assert graphRDD.partitioner().get().equals(newViewIncomingRDD.partitioner().get());
newViewIncomingRDD
    .foreachPartition(partitionIterator -> {

内容来源于网络，如有侵权，请联系作者删除！

JavaPairRDD

相关文章

    查看更多
    微信公众号

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多

    JavaPairRDD类方法

    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com