org.apache.flink.api.common.operators.Operator.getParallelism()方法的使用及代码示例

x33g5p2x  于2022-01-26 转载在 其他  
字(13.7k)|赞(0)|评价(0)|浏览(85)

本文整理了Java中org.apache.flink.api.common.operators.Operator.getParallelism()方法的一些代码示例，展示了Operator.getParallelism()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。Operator.getParallelism()方法的具体详情如下：
包路径：org.apache.flink.api.common.operators.Operator
类名称：Operator
方法名：getParallelism

Operator.getParallelism介绍

[英]Gets the parallelism for this contract instance. The parallelism denotes how many parallel instances of the user function will be spawned during the execution. If this value is ExecutionConfig#PARALLELISM_DEFAULT, then the system will decide the number of parallel instances by itself.
[中]获取此合约实例的并行性。并行性表示在执行过程中将产生多少个用户函数的并行实例。如果该值为ExecutionConfig#PARALLELISM_DEFAULT，则系统将自行决定并行实例的数量。

代码示例

代码示例来源：origin: apache/flink

@Override
public boolean preVisit(Operator<?> visitable) {
  if (!visitedOperators.add(visitable)) {
    return false;
  }
  this.maxDop = Math.max(this.maxDop, visitable.getParallelism());
  return true;
}

代码示例来源：origin: apache/flink

if (this.getParallelism() < 0) {
  noop.setParallelism(input.getParallelism());
} else {

代码示例来源：origin: apache/flink

public static <T, K> org.apache.flink.api.common.operators.SingleInputOperator<?, T, ?> appendKeyRemover(
    org.apache.flink.api.common.operators.Operator<Tuple2<K, T>> inputWithKey,
    SelectorFunctionKeys<T, K> key) {
  TypeInformation<T> inputType = key.getInputType();
  TypeInformation<Tuple2<K, T>> typeInfoWithKey = createTypeWithKey(key);
  MapOperatorBase<Tuple2<K, T>, T, MapFunction<Tuple2<K, T>, T>> mapper =
      new MapOperatorBase<Tuple2<K, T>, T, MapFunction<Tuple2<K, T>, T>>(
          new KeyRemovingMapper<T, K>(),
          new UnaryOperatorInformation<>(typeInfoWithKey, inputType),
          "Key Remover"
      );
  mapper.setInput(inputWithKey);
  mapper.setParallelism(inputWithKey.getParallelism());
  return mapper;
}

代码示例来源：origin: apache/flink

@Override
  protected org.apache.flink.api.common.operators.base.FilterOperatorBase<T, FlatMapFunction<T, T>> translateToDataFlow(Operator<T> input) {

    String name = getName() != null ? getName() : "Filter at " + defaultName;

    // create operator
    PlanFilterOperator<T> po = new PlanFilterOperator<T>(function, name, getInputType());
    po.setInput(input);

    // set parallelism
    if (getParallelism() > 0) {
      // use specified parallelism
      po.setParallelism(getParallelism());
    } else {
      // if no parallelism has been specified, use parallelism of input operator to enable chaining
      po.setParallelism(input.getParallelism());
    }

    return po;
  }
}

代码示例来源：origin: apache/flink

@Override
  protected MapPartitionOperatorBase<IN, OUT, MapPartitionFunction<IN, OUT>> translateToDataFlow(Operator<IN> input) {

    String name = getName() != null ? getName() : "MapPartition at " + defaultName;
    // create operator
    MapPartitionOperatorBase<IN, OUT, MapPartitionFunction<IN, OUT>> po = new MapPartitionOperatorBase<IN, OUT, MapPartitionFunction<IN, OUT>>(function, new UnaryOperatorInformation<IN, OUT>(getInputType(), getResultType()), name);
    // set input
    po.setInput(input);
    // set parallelism
    if (this.getParallelism() > 0) {
      // use specified parallelism
      po.setParallelism(this.getParallelism());
    } else {
      // if no parallelism has been specified, use parallelism of input operator to enable chaining
      po.setParallelism(input.getParallelism());
    }

    return po;
  }
}

代码示例来源：origin: apache/flink

protected GenericDataSinkBase<T> translateToDataFlow(Operator<T> input) {
  // select the name (or create a default one)
  String name = this.name != null ? this.name : this.format.toString();
  GenericDataSinkBase<T> sink = new GenericDataSinkBase<>(this.format, new UnaryOperatorInformation<>(this.type, new NothingTypeInfo()), name);
  // set input
  sink.setInput(input);
  // set parameters
  if (this.parameters != null) {
    sink.getParameters().addAll(this.parameters);
  }
  // set parallelism
  if (this.parallelism > 0) {
    // use specified parallelism
    sink.setParallelism(this.parallelism);
  } else {
    // if no parallelism has been specified, use parallelism of input operator to enable chaining
    sink.setParallelism(input.getParallelism());
  }
  if (this.sortKeyPositions != null) {
    // configure output sorting
    Ordering ordering = new Ordering();
    for (int i = 0; i < this.sortKeyPositions.length; i++) {
      ordering.appendOrdering(this.sortKeyPositions[i], null, this.sortOrders[i]);
    }
    sink.setLocalOrder(ordering);
  }
  return sink;
}

代码示例来源：origin: apache/flink

@Override
protected MapOperatorBase<IN, OUT, MapFunction<IN, OUT>> translateToDataFlow(Operator<IN> input) {
  String name = getName() != null ? getName() : "Map at " + defaultName;
  // create operator
  MapOperatorBase<IN, OUT, MapFunction<IN, OUT>> po = new MapOperatorBase<IN, OUT, MapFunction<IN, OUT>>(function,
      new UnaryOperatorInformation<IN, OUT>(getInputType(), getResultType()), name);
  // set input
  po.setInput(input);
  // set parallelism
  if (this.getParallelism() > 0) {
    // use specified parallelism
    po.setParallelism(this.getParallelism());
  } else {
    // if no parallelism has been specified, use parallelism of input operator to enable chaining
    po.setParallelism(input.getParallelism());
  }
  return po;
}

代码示例来源：origin: apache/flink

@Override
  protected FlatMapOperatorBase<IN, OUT, FlatMapFunction<IN, OUT>> translateToDataFlow(Operator<IN> input) {
    String name = getName() != null ? getName() : "FlatMap at " + defaultName;
    // create operator
    FlatMapOperatorBase<IN, OUT, FlatMapFunction<IN, OUT>> po = new FlatMapOperatorBase<IN, OUT, FlatMapFunction<IN, OUT>>(function,
      new UnaryOperatorInformation<IN, OUT>(getInputType(), getResultType()), name);
    // set input
    po.setInput(input);
    // set parallelism
    if (this.getParallelism() > 0) {
      // use specified parallelism
      po.setParallelism(this.getParallelism());
    } else {
      // if no parallelism has been specified, use parallelism of input operator to enable chaining
      po.setParallelism(input.getParallelism());
    }

    return po;
  }
}

代码示例来源：origin: apache/flink

private <K> org.apache.flink.api.common.operators.SingleInputOperator<?, T, ?> translateToDataFlowWithKeyExtractor(
  Operator<T> input, Keys.SelectorFunctionKeys<T, K> keys, Order order, String name) {
  TypeInformation<Tuple2<K, T>> typeInfoWithKey = KeyFunctions.createTypeWithKey(keys);
  Keys.ExpressionKeys<Tuple2<K, T>> newKey = new Keys.ExpressionKeys<>(0, typeInfoWithKey);
  Operator<Tuple2<K, T>> keyedInput = KeyFunctions.appendKeyExtractor(input, keys);
  int[] sortKeyPositions = newKey.computeLogicalKeyPositions();
  Ordering partitionOrdering = new Ordering();
  for (int keyPosition : sortKeyPositions) {
    partitionOrdering.appendOrdering(keyPosition, null, order);
  }
  // distinguish between partition types
  UnaryOperatorInformation<Tuple2<K, T>, Tuple2<K, T>> operatorInfo = new UnaryOperatorInformation<>(typeInfoWithKey, typeInfoWithKey);
  SortPartitionOperatorBase<Tuple2<K, T>> noop = new SortPartitionOperatorBase<>(operatorInfo, partitionOrdering, name);
  noop.setInput(keyedInput);
  if (this.getParallelism() < 0) {
    // use parallelism of input if not explicitly specified
    noop.setParallelism(input.getParallelism());
  } else {
    // use explicitly specified parallelism
    noop.setParallelism(this.getParallelism());
  }
  return KeyFunctions.appendKeyRemover(noop, keys);
}

代码示例来源：origin: apache/flink

int par = c.getParallelism();
if (n instanceof BinaryUnionNode) {

代码示例来源：origin: apache/flink

assertEquals(unionOperator.getFirstInput().getParallelism(), 3);
assertEquals(unionOperator.getSecondInput().getParallelism(), 2);

代码示例来源：origin: apache/flink

@SuppressWarnings("unchecked")
public static <T, K> org.apache.flink.api.common.operators.Operator<Tuple2<K, T>> appendKeyExtractor(
    org.apache.flink.api.common.operators.Operator<T> input,
    SelectorFunctionKeys<T, K> key) {
  if (input instanceof Union) {
    // if input is a union, we apply the key extractors recursively to all inputs
    org.apache.flink.api.common.operators.Operator<T> firstInput = ((Union) input).getFirstInput();
    org.apache.flink.api.common.operators.Operator<T> secondInput = ((Union) input).getSecondInput();
    org.apache.flink.api.common.operators.Operator<Tuple2<K, T>> firstInputWithKey =
        appendKeyExtractor(firstInput, key);
    org.apache.flink.api.common.operators.Operator<Tuple2<K, T>> secondInputWithKey =
        appendKeyExtractor(secondInput, key);
    return new Union(firstInputWithKey, secondInputWithKey, input.getName());
  }
  TypeInformation<T> inputType = key.getInputType();
  TypeInformation<Tuple2<K, T>> typeInfoWithKey = createTypeWithKey(key);
  KeyExtractingMapper<T, K> extractor = new KeyExtractingMapper(key.getKeyExtractor());
  MapOperatorBase<T, Tuple2<K, T>, MapFunction<T, Tuple2<K, T>>> mapper =
      new MapOperatorBase<T, Tuple2<K, T>, MapFunction<T, Tuple2<K, T>>>(
          extractor,
          new UnaryOperatorInformation(inputType, typeInfoWithKey),
          "Key Extractor"
      );
  mapper.setInput(input);
  mapper.setParallelism(input.getParallelism());
  return mapper;
}

代码示例来源：origin: apache/flink

assertEquals(firstUnionOperator.getFirstInput().getParallelism(), 2);
assertEquals(firstUnionOperator.getSecondInput().getParallelism(), 3);
assertEquals(secondUnionOperator.getSecondInput().getParallelism(), -1);

代码示例来源：origin: apache/flink

@SuppressWarnings("unchecked")
public static <T, K1, K2> org.apache.flink.api.common.operators.Operator<Tuple3<K1, K2, T>> appendKeyExtractor(
    org.apache.flink.api.common.operators.Operator<T> input,
    SelectorFunctionKeys<T, K1> key1,
    SelectorFunctionKeys<T, K2> key2) {
  if (input instanceof Union) {
    // if input is a union, we apply the key extractors recursively to all inputs
    org.apache.flink.api.common.operators.Operator<T> firstInput = ((Union) input).getFirstInput();
    org.apache.flink.api.common.operators.Operator<T> secondInput = ((Union) input).getSecondInput();
    org.apache.flink.api.common.operators.Operator<Tuple3<K1, K2, T>> firstInputWithKey =
        appendKeyExtractor(firstInput, key1, key2);
    org.apache.flink.api.common.operators.Operator<Tuple3<K1, K2, T>> secondInputWithKey =
        appendKeyExtractor(secondInput, key1, key2);
    return new Union(firstInputWithKey, secondInputWithKey, input.getName());
  }
  TypeInformation<T> inputType = key1.getInputType();
  TypeInformation<Tuple3<K1, K2, T>> typeInfoWithKey = createTypeWithKey(key1, key2);
  TwoKeyExtractingMapper<T, K1, K2> extractor =
      new TwoKeyExtractingMapper<>(key1.getKeyExtractor(), key2.getKeyExtractor());
  MapOperatorBase<T, Tuple3<K1, K2, T>, MapFunction<T, Tuple3<K1, K2, T>>> mapper =
      new MapOperatorBase<T, Tuple3<K1, K2, T>, MapFunction<T, Tuple3<K1, K2, T>>>(
          extractor,
          new UnaryOperatorInformation<>(inputType, typeInfoWithKey),
          "Key Extractor"
      );
  mapper.setInput(input);
  mapper.setParallelism(input.getParallelism());
  return mapper;
}

代码示例来源：origin: com.alibaba.blink/flink-core

@Override
public boolean preVisit(Operator<?> visitable) {
  this.maxDop = Math.max(this.maxDop, visitable.getParallelism());
  return true;
}

代码示例来源：origin: org.apache.flink/flink-core

@Override
public boolean preVisit(Operator<?> visitable) {
  if (!visitedOperators.add(visitable)) {
    return false;
  }
  this.maxDop = Math.max(this.maxDop, visitable.getParallelism());
  return true;
}

代码示例来源：origin: com.alibaba.blink/flink-java

public static <T, K> org.apache.flink.api.common.operators.SingleInputOperator<?, T, ?> appendKeyRemover(
    org.apache.flink.api.common.operators.Operator<Tuple2<K, T>> inputWithKey,
    SelectorFunctionKeys<T, K> key) {
  TypeInformation<T> inputType = key.getInputType();
  TypeInformation<Tuple2<K, T>> typeInfoWithKey = createTypeWithKey(key);
  MapOperatorBase<Tuple2<K, T>, T, MapFunction<Tuple2<K, T>, T>> mapper =
      new MapOperatorBase<Tuple2<K, T>, T, MapFunction<Tuple2<K, T>, T>>(
          new KeyRemovingMapper<T, K>(),
          new UnaryOperatorInformation<>(typeInfoWithKey, inputType),
          "Key Remover"
      );
  mapper.setInput(inputWithKey);
  mapper.setParallelism(inputWithKey.getParallelism());
  return mapper;
}

代码示例来源：origin: org.apache.flink/flink-java

public static <T, K> org.apache.flink.api.common.operators.SingleInputOperator<?, T, ?> appendKeyRemover(
    org.apache.flink.api.common.operators.Operator<Tuple2<K, T>> inputWithKey,
    SelectorFunctionKeys<T, K> key) {
  TypeInformation<T> inputType = key.getInputType();
  TypeInformation<Tuple2<K, T>> typeInfoWithKey = createTypeWithKey(key);
  MapOperatorBase<Tuple2<K, T>, T, MapFunction<Tuple2<K, T>, T>> mapper =
      new MapOperatorBase<Tuple2<K, T>, T, MapFunction<Tuple2<K, T>, T>>(
          new KeyRemovingMapper<T, K>(),
          new UnaryOperatorInformation<>(typeInfoWithKey, inputType),
          "Key Remover"
      );
  mapper.setInput(inputWithKey);
  mapper.setParallelism(inputWithKey.getParallelism());
  return mapper;
}

代码示例来源：origin: org.apache.flink/flink-java

@Override
  protected org.apache.flink.api.common.operators.base.FilterOperatorBase<T, FlatMapFunction<T, T>> translateToDataFlow(Operator<T> input) {

    String name = getName() != null ? getName() : "Filter at " + defaultName;

    // create operator
    PlanFilterOperator<T> po = new PlanFilterOperator<T>(function, name, getInputType());
    po.setInput(input);

    // set parallelism
    if (getParallelism() > 0) {
      // use specified parallelism
      po.setParallelism(getParallelism());
    } else {
      // if no parallelism has been specified, use parallelism of input operator to enable chaining
      po.setParallelism(input.getParallelism());
    }

    return po;
  }
}

代码示例来源：origin: com.alibaba.blink/flink-java

@Override
  protected org.apache.flink.api.common.operators.base.FilterOperatorBase<T, FlatMapFunction<T, T>> translateToDataFlow(Operator<T> input) {

    String name = getName() != null ? getName() : "Filter at " + defaultName;

    // create operator
    PlanFilterOperator<T> po = new PlanFilterOperator<T>(function, name, getInputType());
    po.setInput(input);

    // set parallelism
    if (getParallelism() > 0) {
      // use specified parallelism
      po.setParallelism(getParallelism());
    } else {
      // if no parallelism has been specified, use parallelism of input operator to enable chaining
      po.setParallelism(input.getParallelism());
    }

    return po;
  }
}

内容来源于网络，如有侵权，请联系作者删除！

Operator

相关文章

    查看更多
    微信公众号

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com