org.apache.spark.api.java.JavaSparkContext.fromSparkContext()方法的使用及代码示例

x33g5p2x  于2022-01-21 转载在 其他  
字(10.7k)|赞(0)|评价(0)|浏览(143)

本文整理了Java中org.apache.spark.api.java.JavaSparkContext.fromSparkContext()方法的一些代码示例，展示了JavaSparkContext.fromSparkContext()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。JavaSparkContext.fromSparkContext()方法的具体详情如下：
包路径：org.apache.spark.api.java.JavaSparkContext
类名称：JavaSparkContext
方法名：fromSparkContext

JavaSparkContext.fromSparkContext介绍

暂无

代码示例

代码示例来源：origin: OryxProject/oryx

protected final JavaStreamingContext buildStreamingContext() {
 log.info("Starting SparkContext with interval {} seconds", generationIntervalSec);
 SparkConf sparkConf = new SparkConf();
 // Only for tests, really
 if (sparkConf.getOption("spark.master").isEmpty()) {
  log.info("Overriding master to {} for tests", streamingMaster);
  sparkConf.setMaster(streamingMaster);
 }
 // Only for tests, really
 if (sparkConf.getOption("spark.app.name").isEmpty()) {
  String appName = "Oryx" + getLayerName();
  if (id != null) {
   appName = appName + "-" + id;
  }
  log.info("Overriding app name to {} for tests", appName);
  sparkConf.setAppName(appName);
 }
 extraSparkConfig.forEach((key, value) -> sparkConf.setIfMissing(key, value.toString()));
 // Turn this down to prevent long blocking at shutdown
 sparkConf.setIfMissing(
   "spark.streaming.gracefulStopTimeout",
   Long.toString(TimeUnit.MILLISECONDS.convert(generationIntervalSec, TimeUnit.SECONDS)));
 sparkConf.setIfMissing("spark.cleaner.ttl", Integer.toString(20 * generationIntervalSec));
 long generationIntervalMS =
   TimeUnit.MILLISECONDS.convert(generationIntervalSec, TimeUnit.SECONDS);
 JavaSparkContext jsc = JavaSparkContext.fromSparkContext(SparkContext.getOrCreate(sparkConf));
 return new JavaStreamingContext(jsc, new Duration(generationIntervalMS));
}

代码示例来源：origin: OryxProject/oryx

@BeforeClass
public static void setUp() {
 SparkConf sparkConf = new SparkConf().setMaster("local[*]").setAppName("SparkIT");
 javaSparkContext = JavaSparkContext.fromSparkContext(SparkContext.getOrCreate(sparkConf));
}

代码示例来源：origin: mahmoudparsian/data-algorithms-book

.getOrCreate();
JavaSparkContext context = JavaSparkContext.fromSparkContext(session.sparkContext());

代码示例来源：origin: mahmoudparsian/data-algorithms-book

.getOrCreate();
JavaSparkContext context = JavaSparkContext.fromSparkContext(session.sparkContext());

代码示例来源：origin: mahmoudparsian/data-algorithms-book

.getOrCreate();
JavaSparkContext context = JavaSparkContext.fromSparkContext(session.sparkContext());

代码示例来源：origin: locationtech/geowave

private void initContext() {
 if (session == null) {
  String jar = "";
  try {
   jar =
     KMeansRunner.class.getProtectionDomain().getCodeSource().getLocation().toURI().getPath();
  } catch (final URISyntaxException e) {
   LOGGER.error("Unable to set jar location in spark configuration", e);
  }
  session = GeoWaveSparkConf.createSessionFromParams(appName, master, host, jar);
  jsc = JavaSparkContext.fromSparkContext(session.sparkContext());
 }
}

代码示例来源：origin: co.cask.cdap/hydrator-spark-core2

@Override
public JavaRDD<U> transform(SparkExecutionPluginContext context, JavaRDD<T> input) throws Exception {
 lazyInit(JavaSparkContext.fromSparkContext(input.context()));
 return delegate.transform(context, input);
}

代码示例来源：origin: com.cloudera.oryx/oryx-lambda

protected final JavaStreamingContext buildStreamingContext() {
 log.info("Starting SparkContext with interval {} seconds", generationIntervalSec);
 SparkConf sparkConf = new SparkConf();
 // Only for tests, really
 if (sparkConf.getOption("spark.master").isEmpty()) {
  log.info("Overriding master to {} for tests", streamingMaster);
  sparkConf.setMaster(streamingMaster);
 }
 // Only for tests, really
 if (sparkConf.getOption("spark.app.name").isEmpty()) {
  String appName = "Oryx" + getLayerName();
  if (id != null) {
   appName = appName + "-" + id;
  }
  log.info("Overriding app name to {} for tests", appName);
  sparkConf.setAppName(appName);
 }
 extraSparkConfig.forEach((key, value) -> sparkConf.setIfMissing(key, value.toString()));
 // Turn this down to prevent long blocking at shutdown
 sparkConf.setIfMissing(
   "spark.streaming.gracefulStopTimeout",
   Long.toString(TimeUnit.MILLISECONDS.convert(generationIntervalSec, TimeUnit.SECONDS)));
 sparkConf.setIfMissing("spark.cleaner.ttl", Integer.toString(20 * generationIntervalSec));
 long generationIntervalMS =
   TimeUnit.MILLISECONDS.convert(generationIntervalSec, TimeUnit.SECONDS);
 JavaSparkContext jsc = JavaSparkContext.fromSparkContext(SparkContext.getOrCreate(sparkConf));
 return new JavaStreamingContext(jsc, new Duration(generationIntervalMS));
}

代码示例来源：origin: co.cask.cdap/hydrator-spark-core2

@Override
 public void run(DatasetContext datasetContext) throws Exception {
  PipelineRuntime pipelineRuntime = new SparkPipelineRuntime(sec);
  SparkExecutionPluginContext sparkPluginContext =
   new BasicSparkExecutionPluginContext(sec, JavaSparkContext.fromSparkContext(stream.context().sparkContext()),
                      datasetContext, pipelineRuntime, stageSpec);
  wrappedCompute.initialize(sparkPluginContext);
 }
}, Exception.class);

代码示例来源：origin: locationtech/geowave

Integer.toString(numCores)).getOrCreate();
jsc = JavaSparkContext.fromSparkContext(session.sparkContext());

代码示例来源：origin: uber/uberscriptquery

public static Dataset<Row> readJdbc(String jdbcUrlWithPassword, String sql, SparkSession spark) {
  JavaRDD<Tuple2<List<Row>, StructType>> javaRdd1 = JavaSparkContext.fromSparkContext(spark.sparkContext())
      .parallelize(Arrays.asList(0))
      .map(new Function<Integer, Tuple2<List<Row>, StructType>>() {
        @Override
        public Tuple2<List<Row>, StructType> call(Integer v1) throws Exception {
          Tuple2<List<Row>, StructType> tuple = new ExponentialBackoffRetryPolicy<Tuple2<List<Row>, StructType>>(3, 100)
              .attempt(() -> readJdbcAndReturnRowsAndSchema(jdbcUrlWithPassword, sql));
          return tuple;
        }
      });
  javaRdd1 = javaRdd1.persist(StorageLevel.DISK_ONLY_2());
  StructType structType = javaRdd1.map(new Function<Tuple2<List<Row>, StructType>, StructType>() {
    @Override
    public StructType call(Tuple2<List<Row>, StructType> v1) throws Exception {
      return v1._2();
    }
  }).collect().get(0);
  JavaRDD<Row> javaRdd2 = javaRdd1.flatMap(new FlatMapFunction<Tuple2<List<Row>, StructType>, Row>() {
    @Override
    public Iterator<Row> call(Tuple2<List<Row>, StructType> listStructTypeTuple2) throws Exception {
      return listStructTypeTuple2._1().iterator();
    }
  });
  return spark.createDataFrame(javaRdd2, structType);
  //SqlUtils.loadJdbcDriverClass(jdbcUrlWithPassword);
  //DriverRegistryWrapper.register(com.mysql.jdbc.Driver.class.getName());
  //ExponentialBackoffRetryPolicy<Dataset<Row>> retryPolicy = new ExponentialBackoffRetryPolicy<>(3, 100);
  //return retryPolicy.attempt(() -> readJdbcWithoutRetry(jdbcUrlWithPassword, sql, spark));
}

代码示例来源：origin: uber/marmaray

@NonNull final ErrorExtractor errorExtractor) {
JavaSparkContext jsc = JavaSparkContext.fromSparkContext(sc);
final ErrorTableConfiguration errorTableConf = new ErrorTableConfiguration(conf);
if (!errorTableConf.isEnabled()) {

代码示例来源：origin: co.cask.cdap/hydrator-spark-core2

@Override
 public JavaRDD<U> call(JavaRDD<T> data, Time batchTime) throws Exception {
  SparkExecutionPluginContext sparkPluginContext =
   new SparkStreamingExecutionContext(sec, JavaSparkContext.fromSparkContext(data.context()),
                     batchTime.milliseconds(), stageSpec);
  String stageName = stageSpec.getName();
  data = data.map(new CountingFunction<T>(stageName, sec.getMetrics(), "records.in", null));
  return compute.transform(sparkPluginContext, data)
   .map(new CountingFunction<U>(stageName, sec.getMetrics(), "records.out", sec.getDataTracer(stageName)));
 }
}

代码示例来源：origin: uk.gov.gchq.gaffer/spark-accumulo-library

private JavaRDD<Element> doOperation(final GetJavaRDDOfElements operation,
                   final Context context,
                   final AccumuloStore accumuloStore) throws OperationException {
  final JavaSparkContext sparkContext = JavaSparkContext.fromSparkContext(SparkContextUtil.getSparkSession(context, accumuloStore.getProperties()).sparkContext());
  final Configuration conf = getConfiguration(operation);
  // Use batch scan option when performing seeded operation
  InputConfigurator.setBatchScan(AccumuloInputFormat.class, conf, true);
  addIterators(accumuloStore, conf, context.getUser(), operation);
  addRanges(accumuloStore, conf, operation);
  final JavaPairRDD<Element, NullWritable> pairRDD = sparkContext.newAPIHadoopRDD(conf,
      ElementInputFormat.class,
      Element.class,
      NullWritable.class);
  final JavaRDD<Element> rdd = pairRDD.map(new FirstElement());
  return rdd;
}

代码示例来源：origin: uk.gov.gchq.gaffer/spark-accumulo-library

private JavaRDD<Element> doOperation(final GetJavaRDDOfElementsInRanges operation,
                   final Context context,
                   final AccumuloStore accumuloStore) throws OperationException {
  final JavaSparkContext sparkContext = JavaSparkContext.fromSparkContext(SparkContextUtil.getSparkSession(context, accumuloStore.getProperties()).sparkContext());
  final Configuration conf = getConfiguration(operation);
  // Use batch scan option when performing seeded operation
  InputConfigurator.setBatchScan(AccumuloInputFormat.class, conf, true);
  addIterators(accumuloStore, conf, context.getUser(), operation);
  addRangesFromPairs(accumuloStore, conf, operation);
  final JavaPairRDD<Element, NullWritable> pairRDD = sparkContext.newAPIHadoopRDD(conf,
      ElementInputFormat.class,
      Element.class,
      NullWritable.class);
  final JavaRDD<Element> rdd = pairRDD.map(new FirstElement());
  return rdd;
}

代码示例来源：origin: uk.gov.gchq.gaffer/spark-accumulo-library

public void doOperation(final ImportJavaRDDOfElements operation, final Context context, final AccumuloStore store) throws OperationException {
    final String outputPath = operation.getOption(OUTPUT_PATH);
    if (null == outputPath || outputPath.isEmpty()) {
      throw new OperationException("Option outputPath must be set for this option to be run against the accumulostore");
    }
    final String failurePath = operation.getOption(FAILURE_PATH);
    if (null == failurePath || failurePath.isEmpty()) {
      throw new OperationException("Option failurePath must be set for this option to be run against the accumulostore");
    }

    final SparkContext sparkContext = SparkContextUtil.getSparkSession(context, store.getProperties()).sparkContext();
    final Broadcast<AccumuloElementConverter> broadcast = JavaSparkContext.fromSparkContext(sparkContext).broadcast(store.getKeyPackage().getKeyConverter());
    final ElementConverterFunction func = new ElementConverterFunction(broadcast);
    final JavaPairRDD<Key, Value> rdd = operation.getInput().flatMapToPair(func);
    final ImportKeyValueJavaPairRDDToAccumulo op =
        new ImportKeyValueJavaPairRDDToAccumulo.Builder()
            .input(rdd)
            .failurePath(failurePath)
            .outputPath(outputPath)
            .build();
    store.execute(new OperationChain(op), context);
  }
}

代码示例来源：origin: locationtech/geowave

final JavaSparkContext javaSC = JavaSparkContext.fromSparkContext(sc);

代码示例来源：origin: co.cask.cdap/hydrator-spark-core2

= new SparkStreamingExecutionContext(sec, JavaSparkContext.fromSparkContext(data.rdd().context()),
                    logicalStartTime, stageSpec);
final JavaRDD<T> countedRDD = data.map(new CountingFunction<T>(stageName, sec.getMetrics(),

内容来源于网络，如有侵权，请联系作者删除！

JavaSparkContext

相关文章

    查看更多
    微信公众号

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多

    JavaSparkContext类方法

    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com