org.apache.flink.api.java.operators.GroupReduceOperator.<init>()方法的使用及代码示例

x33g5p2x  于2022-01-20 转载在 其他  
字(15.7k)|赞(0)|评价(0)|浏览(134)

本文整理了Java中org.apache.flink.api.java.operators.GroupReduceOperator.<init>()方法的一些代码示例,展示了GroupReduceOperator.<init>()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。GroupReduceOperator.<init>()方法的具体详情如下:
包路径:org.apache.flink.api.java.operators.GroupReduceOperator
类名称:GroupReduceOperator
方法名:<init>

GroupReduceOperator.<init>介绍

[英]Constructor for a non-grouped reduce (all reduce).
[中]非分组reduce(所有reduce)的构造函数。

代码示例

代码示例来源:origin: apache/flink

/**
 * Applies a GroupReduce transformation on a non-grouped {@link DataSet}.
 *
 * <p>The transformation calls a {@link org.apache.flink.api.common.functions.RichGroupReduceFunction} once with the full DataSet.
 * The GroupReduceFunction can iterate over all elements of the DataSet and emit any
 *   number of output elements including none.
 *
 * @param reducer The GroupReduceFunction that is applied on the DataSet.
 * @return A GroupReduceOperator that represents the reduced DataSet.
 *
 * @see org.apache.flink.api.common.functions.RichGroupReduceFunction
 * @see org.apache.flink.api.java.operators.GroupReduceOperator
 * @see DataSet
 */
public <R> GroupReduceOperator<T, R> reduceGroup(GroupReduceFunction<T, R> reducer) {
  if (reducer == null) {
    throw new NullPointerException("GroupReduce function must not be null.");
  }
  String callLocation = Utils.getCallLocationName();
  TypeInformation<R> resultType = TypeExtractor.getGroupReduceReturnTypes(reducer, getType(), callLocation, true);
  return new GroupReduceOperator<>(this, resultType, clean(reducer), callLocation);
}

代码示例来源:origin: apache/flink

/**
 * Applies a GroupReduce transformation on a grouped and sorted {@link DataSet}.
 *
 * <p>The transformation calls a {@link org.apache.flink.api.common.functions.RichGroupReduceFunction} for each group of the DataSet.
 * A GroupReduceFunction can iterate over all elements of a group and emit any
 *   number of output elements including none.
 *
 * @param reducer The GroupReduceFunction that is applied on each group of the DataSet.
 * @return A GroupReduceOperator that represents the reduced DataSet.
 *
 * @see org.apache.flink.api.common.functions.RichGroupReduceFunction
 * @see GroupReduceOperator
 * @see DataSet
 */
public <R> GroupReduceOperator<T, R> reduceGroup(GroupReduceFunction<T, R> reducer) {
  if (reducer == null) {
    throw new NullPointerException("GroupReduce function must not be null.");
  }
  TypeInformation<R> resultType = TypeExtractor.getGroupReduceReturnTypes(reducer,
      inputDataSet.getType(), Utils.getCallLocationName(), true);
  return new GroupReduceOperator<>(this, resultType, inputDataSet.clean(reducer), Utils.getCallLocationName());
}

代码示例来源:origin: apache/flink

/**
 * Applies a GroupReduce transformation on a grouped {@link DataSet}.
 *
 * <p>The transformation calls a {@link org.apache.flink.api.common.functions.RichGroupReduceFunction} for each group of the DataSet.
 * A GroupReduceFunction can iterate over all elements of a group and emit any
 *   number of output elements including none.
 *
 * @param reducer The GroupReduceFunction that is applied on each group of the DataSet.
 * @return A GroupReduceOperator that represents the reduced DataSet.
 *
 * @see org.apache.flink.api.common.functions.RichGroupReduceFunction
 * @see GroupReduceOperator
 * @see DataSet
 */
public <R> GroupReduceOperator<T, R> reduceGroup(GroupReduceFunction<T, R> reducer) {
  if (reducer == null) {
    throw new NullPointerException("GroupReduce function must not be null.");
  }
  TypeInformation<R> resultType = TypeExtractor.getGroupReduceReturnTypes(reducer,
      this.getInputDataSet().getType(), Utils.getCallLocationName(), true);
  return new GroupReduceOperator<T, R>(this, resultType, inputDataSet.clean(reducer), Utils.getCallLocationName());
}

代码示例来源:origin: apache/flink

/**
 * Generate a sample of DataSet which contains fixed size elements.
 *
 * <p><strong>NOTE:</strong> Sample with fixed size is not as efficient as sample with fraction, use sample with
 * fraction unless you need exact precision.
 *
 * @param withReplacement Whether element can be selected more than once.
 * @param numSamples       The expected sample size.
 * @param seed            Random number generator seed.
 * @return The sampled DataSet
 */
public static <T> DataSet<T> sampleWithSize(
  DataSet <T> input,
  final boolean withReplacement,
  final int numSamples,
  final long seed) {
  SampleInPartition<T> sampleInPartition = new SampleInPartition<>(withReplacement, numSamples, seed);
  MapPartitionOperator mapPartitionOperator = input.mapPartition(sampleInPartition);
  // There is no previous group, so the parallelism of GroupReduceOperator is always 1.
  String callLocation = Utils.getCallLocationName();
  SampleInCoordinator<T> sampleInCoordinator = new SampleInCoordinator<>(withReplacement, numSamples, seed);
  return new GroupReduceOperator<>(mapPartitionOperator, input.getType(), sampleInCoordinator, callLocation);
}

代码示例来源:origin: org.apache.flink/flink-java

/**
 * Applies a GroupReduce transformation on a non-grouped {@link DataSet}.
 *
 * <p>The transformation calls a {@link org.apache.flink.api.common.functions.RichGroupReduceFunction} once with the full DataSet.
 * The GroupReduceFunction can iterate over all elements of the DataSet and emit any
 *   number of output elements including none.
 *
 * @param reducer The GroupReduceFunction that is applied on the DataSet.
 * @return A GroupReduceOperator that represents the reduced DataSet.
 *
 * @see org.apache.flink.api.common.functions.RichGroupReduceFunction
 * @see org.apache.flink.api.java.operators.GroupReduceOperator
 * @see DataSet
 */
public <R> GroupReduceOperator<T, R> reduceGroup(GroupReduceFunction<T, R> reducer) {
  if (reducer == null) {
    throw new NullPointerException("GroupReduce function must not be null.");
  }
  String callLocation = Utils.getCallLocationName();
  TypeInformation<R> resultType = TypeExtractor.getGroupReduceReturnTypes(reducer, getType(), callLocation, true);
  return new GroupReduceOperator<>(this, resultType, clean(reducer), callLocation);
}

代码示例来源:origin: com.alibaba.blink/flink-java

/**
 * Applies a GroupReduce transformation on a non-grouped {@link DataSet}.
 *
 * <p>The transformation calls a {@link org.apache.flink.api.common.functions.RichGroupReduceFunction} once with the full DataSet.
 * The GroupReduceFunction can iterate over all elements of the DataSet and emit any
 *   number of output elements including none.
 *
 * @param reducer The GroupReduceFunction that is applied on the DataSet.
 * @return A GroupReduceOperator that represents the reduced DataSet.
 *
 * @see org.apache.flink.api.common.functions.RichGroupReduceFunction
 * @see org.apache.flink.api.java.operators.GroupReduceOperator
 * @see DataSet
 */
public <R> GroupReduceOperator<T, R> reduceGroup(GroupReduceFunction<T, R> reducer) {
  if (reducer == null) {
    throw new NullPointerException("GroupReduce function must not be null.");
  }
  String callLocation = Utils.getCallLocationName();
  TypeInformation<R> resultType = TypeExtractor.getGroupReduceReturnTypes(reducer, getType(), callLocation, true);
  return new GroupReduceOperator<>(this, resultType, clean(reducer), callLocation);
}

代码示例来源:origin: org.apache.flink/flink-java

/**
 * Applies a GroupReduce transformation on a grouped and sorted {@link DataSet}.
 *
 * <p>The transformation calls a {@link org.apache.flink.api.common.functions.RichGroupReduceFunction} for each group of the DataSet.
 * A GroupReduceFunction can iterate over all elements of a group and emit any
 *   number of output elements including none.
 *
 * @param reducer The GroupReduceFunction that is applied on each group of the DataSet.
 * @return A GroupReduceOperator that represents the reduced DataSet.
 *
 * @see org.apache.flink.api.common.functions.RichGroupReduceFunction
 * @see GroupReduceOperator
 * @see DataSet
 */
public <R> GroupReduceOperator<T, R> reduceGroup(GroupReduceFunction<T, R> reducer) {
  if (reducer == null) {
    throw new NullPointerException("GroupReduce function must not be null.");
  }
  TypeInformation<R> resultType = TypeExtractor.getGroupReduceReturnTypes(reducer,
      inputDataSet.getType(), Utils.getCallLocationName(), true);
  return new GroupReduceOperator<>(this, resultType, inputDataSet.clean(reducer), Utils.getCallLocationName());
}

代码示例来源:origin: com.alibaba.blink/flink-java

/**
 * Applies a GroupReduce transformation on a grouped and sorted {@link DataSet}.
 *
 * <p>The transformation calls a {@link org.apache.flink.api.common.functions.RichGroupReduceFunction} for each group of the DataSet.
 * A GroupReduceFunction can iterate over all elements of a group and emit any
 *   number of output elements including none.
 *
 * @param reducer The GroupReduceFunction that is applied on each group of the DataSet.
 * @return A GroupReduceOperator that represents the reduced DataSet.
 *
 * @see org.apache.flink.api.common.functions.RichGroupReduceFunction
 * @see GroupReduceOperator
 * @see DataSet
 */
public <R> GroupReduceOperator<T, R> reduceGroup(GroupReduceFunction<T, R> reducer) {
  if (reducer == null) {
    throw new NullPointerException("GroupReduce function must not be null.");
  }
  TypeInformation<R> resultType = TypeExtractor.getGroupReduceReturnTypes(reducer,
      inputDataSet.getType(), Utils.getCallLocationName(), true);
  return new GroupReduceOperator<>(this, resultType, inputDataSet.clean(reducer), Utils.getCallLocationName());
}

代码示例来源:origin: com.alibaba.blink/flink-java

/**
 * Applies a GroupReduce transformation on a grouped {@link DataSet}.
 *
 * <p>The transformation calls a {@link org.apache.flink.api.common.functions.RichGroupReduceFunction} for each group of the DataSet.
 * A GroupReduceFunction can iterate over all elements of a group and emit any
 *   number of output elements including none.
 *
 * @param reducer The GroupReduceFunction that is applied on each group of the DataSet.
 * @return A GroupReduceOperator that represents the reduced DataSet.
 *
 * @see org.apache.flink.api.common.functions.RichGroupReduceFunction
 * @see GroupReduceOperator
 * @see DataSet
 */
public <R> GroupReduceOperator<T, R> reduceGroup(GroupReduceFunction<T, R> reducer) {
  if (reducer == null) {
    throw new NullPointerException("GroupReduce function must not be null.");
  }
  TypeInformation<R> resultType = TypeExtractor.getGroupReduceReturnTypes(reducer,
      this.getInputDataSet().getType(), Utils.getCallLocationName(), true);
  return new GroupReduceOperator<T, R>(this, resultType, inputDataSet.clean(reducer), Utils.getCallLocationName());
}

代码示例来源:origin: org.apache.flink/flink-java

/**
 * Applies a GroupReduce transformation on a grouped {@link DataSet}.
 *
 * <p>The transformation calls a {@link org.apache.flink.api.common.functions.RichGroupReduceFunction} for each group of the DataSet.
 * A GroupReduceFunction can iterate over all elements of a group and emit any
 *   number of output elements including none.
 *
 * @param reducer The GroupReduceFunction that is applied on each group of the DataSet.
 * @return A GroupReduceOperator that represents the reduced DataSet.
 *
 * @see org.apache.flink.api.common.functions.RichGroupReduceFunction
 * @see GroupReduceOperator
 * @see DataSet
 */
public <R> GroupReduceOperator<T, R> reduceGroup(GroupReduceFunction<T, R> reducer) {
  if (reducer == null) {
    throw new NullPointerException("GroupReduce function must not be null.");
  }
  TypeInformation<R> resultType = TypeExtractor.getGroupReduceReturnTypes(reducer,
      this.getInputDataSet().getType(), Utils.getCallLocationName(), true);
  return new GroupReduceOperator<T, R>(this, resultType, inputDataSet.clean(reducer), Utils.getCallLocationName());
}

代码示例来源:origin: org.apache.flink/flink-java

/**
 * Generate a sample of DataSet which contains fixed size elements.
 *
 * <p><strong>NOTE:</strong> Sample with fixed size is not as efficient as sample with fraction, use sample with
 * fraction unless you need exact precision.
 *
 * @param withReplacement Whether element can be selected more than once.
 * @param numSamples       The expected sample size.
 * @param seed            Random number generator seed.
 * @return The sampled DataSet
 */
public static <T> DataSet<T> sampleWithSize(
  DataSet <T> input,
  final boolean withReplacement,
  final int numSamples,
  final long seed) {
  SampleInPartition<T> sampleInPartition = new SampleInPartition<>(withReplacement, numSamples, seed);
  MapPartitionOperator mapPartitionOperator = input.mapPartition(sampleInPartition);
  // There is no previous group, so the parallelism of GroupReduceOperator is always 1.
  String callLocation = Utils.getCallLocationName();
  SampleInCoordinator<T> sampleInCoordinator = new SampleInCoordinator<>(withReplacement, numSamples, seed);
  return new GroupReduceOperator<>(mapPartitionOperator, input.getType(), sampleInCoordinator, callLocation);
}

代码示例来源:origin: com.alibaba.blink/flink-java

/**
 * Generate a sample of DataSet which contains fixed size elements.
 *
 * <p><strong>NOTE:</strong> Sample with fixed size is not as efficient as sample with fraction, use sample with
 * fraction unless you need exact precision.
 *
 * @param withReplacement Whether element can be selected more than once.
 * @param numSamples       The expected sample size.
 * @param seed            Random number generator seed.
 * @return The sampled DataSet
 */
public static <T> DataSet<T> sampleWithSize(
  DataSet <T> input,
  final boolean withReplacement,
  final int numSamples,
  final long seed) {
  SampleInPartition<T> sampleInPartition = new SampleInPartition<>(withReplacement, numSamples, seed);
  MapPartitionOperator mapPartitionOperator = input.mapPartition(sampleInPartition);
  // There is no previous group, so the parallelism of GroupReduceOperator is always 1.
  String callLocation = Utils.getCallLocationName();
  SampleInCoordinator<T> sampleInCoordinator = new SampleInCoordinator<>(withReplacement, numSamples, seed);
  return new GroupReduceOperator<>(mapPartitionOperator, input.getType(), sampleInCoordinator, callLocation);
}

代码示例来源:origin: org.apache.beam/beam-runners-flink_2.10

new GroupReduceOperator(grouping, typeInformation, doFnWrapper, transform.getName());

代码示例来源:origin: dataArtisans/flink-dataflow

@Override
  public void translateNode(GroupByKey.GroupByKeyOnly<K, V> transform, FlinkBatchTranslationContext context) {
    DataSet<KV<K, V>> inputDataSet = context.getInputDataSet(context.getInput(transform));
    GroupReduceFunction<KV<K, V>, KV<K, Iterable<V>>> groupReduceFunction = new FlinkKeyedListAggregationFunction<>();
    TypeInformation<KV<K, Iterable<V>>> typeInformation = context.getTypeInfo(context.getOutput(transform));
    Grouping<KV<K, V>> grouping = new UnsortedGrouping<>(inputDataSet, new Keys.ExpressionKeys<>(new String[]{"key"}, inputDataSet.getType()));
    GroupReduceOperator<KV<K, V>, KV<K, Iterable<V>>> outputDataSet =
        new GroupReduceOperator<>(grouping, typeInformation, groupReduceFunction, transform.getName());
    context.setOutputDataSet(context.getOutput(transform), outputDataSet);
  }
}

代码示例来源:origin: dataArtisans/flink-dataflow

@Override
  public void translateNode(GroupByKey<K, V> transform, FlinkBatchTranslationContext context) {
    DataSet<KV<K, V>> inputDataSet = context.getInputDataSet(context.getInput(transform));
    GroupReduceFunction<KV<K, V>, KV<K, Iterable<V>>> groupReduceFunction = new FlinkKeyedListAggregationFunction<>();
    TypeInformation<KV<K, Iterable<V>>> typeInformation = context.getTypeInfo(context.getOutput(transform));
    Grouping<KV<K, V>> grouping = new UnsortedGrouping<>(inputDataSet, new Keys.ExpressionKeys<>(new String[]{"key"}, inputDataSet.getType()));
    GroupReduceOperator<KV<K, V>, KV<K, Iterable<V>>> outputDataSet =
        new GroupReduceOperator<>(grouping, typeInformation, groupReduceFunction, transform.getName());
    context.setOutputDataSet(context.getOutput(transform), outputDataSet);
  }
}

代码示例来源:origin: org.apache.beam/beam-runners-flink_2.11

new GroupReduceOperator<>(
  intermediateGrouping, partialReduceTypeInfo, reduceFunction, fullName);

代码示例来源:origin: org.apache.beam/beam-runners-flink

new GroupReduceOperator<>(
  intermediateGrouping, partialReduceTypeInfo, reduceFunction, fullName);

代码示例来源:origin: org.apache.beam/beam-runners-flink_2.10

new GroupReduceOperator<>(
  intermediateGrouping, partialReduceTypeInfo, reduceFunction, transform.getName());

代码示例来源:origin: org.apache.beam/beam-runners-flink_2.10

new GroupReduceOperator<>(
  intermediateGrouping, reduceTypeInfo, reduceFunction, transform.getName());
new GroupReduceOperator<>(
  grouping, reduceTypeInfo, reduceFunction, transform.getName());

代码示例来源:origin: dataArtisans/flink-dataflow

new GroupReduceOperator<>(intermediateGrouping, reduceTypeInfo, reduceFunction, transform.getName());

相关文章