org.apache.spark.api.java.JavaSparkContext.setLocalProperty()方法的使用及代码示例

x33g5p2x  于2022-01-21 转载在 其他  
字(3.8k)|赞(0)|评价(0)|浏览(92)

本文整理了Java中org.apache.spark.api.java.JavaSparkContext.setLocalProperty()方法的一些代码示例,展示了JavaSparkContext.setLocalProperty()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。JavaSparkContext.setLocalProperty()方法的具体详情如下:
包路径:org.apache.spark.api.java.JavaSparkContext
类名称:JavaSparkContext
方法名:setLocalProperty

JavaSparkContext.setLocalProperty介绍

暂无

代码示例

代码示例来源:origin: apache/tinkerpop

/**
 * When using a persistent context the running Context's configuration will override a passed
 * in configuration. Spark allows us to override these inherited properties via
 * SparkContext.setLocalProperty
 */
private void updateLocalConfiguration(final JavaSparkContext sparkContext, final Configuration configuration) {
  /*
   * While we could enumerate over the entire SparkConfiguration and copy into the Thread
   * Local properties of the Spark Context this could cause adverse effects with future
   * versions of Spark. Since the api for setting multiple local properties at once is
   * restricted as private, we will only set those properties we know can effect SparkGraphComputer
   * Execution rather than applying the entire configuration.
   */
  final String[] validPropertyNames = {
      "spark.job.description",
      "spark.jobGroup.id",
      "spark.job.interruptOnCancel",
      "spark.scheduler.pool"
  };
  for (String propertyName : validPropertyNames) {
    String propertyValue = configuration.get(propertyName);
    if (propertyValue != null) {
      this.logger.info("Setting Thread Local SparkContext Property - "
          + propertyName + " : " + propertyValue);
      sparkContext.setLocalProperty(propertyName, configuration.get(propertyName));
    }
  }
}

代码示例来源:origin: ai.grakn/grakn-kb

/**
 * When using a persistent context the running Context's configuration will override a passed
 * in configuration. Spark allows us to override these inherited properties via
 * SparkContext.setLocalProperty
 */
private static void updateLocalConfiguration(final JavaSparkContext sparkContext,
                       final Configuration configuration) {
  /*
   * While we could enumerate over the entire SparkConfiguration and copy into the Thread
   * Local properties of the Spark Context this could cause adverse effects with future
   * versions of Spark. Since the api for setting multiple local properties at once is
   * restricted as private, we will only set those properties we know can effect SparkGraphComputer
   * Execution rather than applying the entire configuration.
   */
  final String[] validPropertyNames = {
      "spark.job.description",
      "spark.jobGroup.id",
      "spark.job.interruptOnCancel",
      "spark.scheduler.pool"
  };
  for (String propertyName : validPropertyNames) {
    String propertyValue = configuration.get(propertyName);
    if (propertyValue != null) {
      LOGGER.info("Setting Thread Local SparkContext Property - "
          + propertyName + " : " + propertyValue);
      sparkContext.setLocalProperty(propertyName, configuration.get(propertyName));
    }
  }
}

代码示例来源:origin: org.apache.tinkerpop/spark-gremlin

/**
 * When using a persistent context the running Context's configuration will override a passed
 * in configuration. Spark allows us to override these inherited properties via
 * SparkContext.setLocalProperty
 */
private void updateLocalConfiguration(final JavaSparkContext sparkContext, final Configuration configuration) {
  /*
   * While we could enumerate over the entire SparkConfiguration and copy into the Thread
   * Local properties of the Spark Context this could cause adverse effects with future
   * versions of Spark. Since the api for setting multiple local properties at once is
   * restricted as private, we will only set those properties we know can effect SparkGraphComputer
   * Execution rather than applying the entire configuration.
   */
  final String[] validPropertyNames = {
      "spark.job.description",
      "spark.jobGroup.id",
      "spark.job.interruptOnCancel",
      "spark.scheduler.pool"
  };
  for (String propertyName : validPropertyNames) {
    String propertyValue = configuration.get(propertyName);
    if (propertyValue != null) {
      this.logger.info("Setting Thread Local SparkContext Property - "
          + propertyName + " : " + propertyValue);
      sparkContext.setLocalProperty(propertyName, configuration.get(propertyName));
    }
  }
}

相关文章

微信公众号

最新文章

更多