本文整理了Java中org.apache.spark.api.java.JavaSparkContext.setLocalProperty()
方法的一些代码示例,展示了JavaSparkContext.setLocalProperty()
的具体用法。这些代码示例主要来源于Github
/Stackoverflow
/Maven
等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。JavaSparkContext.setLocalProperty()
方法的具体详情如下:
包路径:org.apache.spark.api.java.JavaSparkContext
类名称:JavaSparkContext
方法名:setLocalProperty
暂无
代码示例来源:origin: apache/tinkerpop
/**
* When using a persistent context the running Context's configuration will override a passed
* in configuration. Spark allows us to override these inherited properties via
* SparkContext.setLocalProperty
*/
private void updateLocalConfiguration(final JavaSparkContext sparkContext, final Configuration configuration) {
/*
* While we could enumerate over the entire SparkConfiguration and copy into the Thread
* Local properties of the Spark Context this could cause adverse effects with future
* versions of Spark. Since the api for setting multiple local properties at once is
* restricted as private, we will only set those properties we know can effect SparkGraphComputer
* Execution rather than applying the entire configuration.
*/
final String[] validPropertyNames = {
"spark.job.description",
"spark.jobGroup.id",
"spark.job.interruptOnCancel",
"spark.scheduler.pool"
};
for (String propertyName : validPropertyNames) {
String propertyValue = configuration.get(propertyName);
if (propertyValue != null) {
this.logger.info("Setting Thread Local SparkContext Property - "
+ propertyName + " : " + propertyValue);
sparkContext.setLocalProperty(propertyName, configuration.get(propertyName));
}
}
}
代码示例来源:origin: ai.grakn/grakn-kb
/**
* When using a persistent context the running Context's configuration will override a passed
* in configuration. Spark allows us to override these inherited properties via
* SparkContext.setLocalProperty
*/
private static void updateLocalConfiguration(final JavaSparkContext sparkContext,
final Configuration configuration) {
/*
* While we could enumerate over the entire SparkConfiguration and copy into the Thread
* Local properties of the Spark Context this could cause adverse effects with future
* versions of Spark. Since the api for setting multiple local properties at once is
* restricted as private, we will only set those properties we know can effect SparkGraphComputer
* Execution rather than applying the entire configuration.
*/
final String[] validPropertyNames = {
"spark.job.description",
"spark.jobGroup.id",
"spark.job.interruptOnCancel",
"spark.scheduler.pool"
};
for (String propertyName : validPropertyNames) {
String propertyValue = configuration.get(propertyName);
if (propertyValue != null) {
LOGGER.info("Setting Thread Local SparkContext Property - "
+ propertyName + " : " + propertyValue);
sparkContext.setLocalProperty(propertyName, configuration.get(propertyName));
}
}
}
代码示例来源:origin: org.apache.tinkerpop/spark-gremlin
/**
* When using a persistent context the running Context's configuration will override a passed
* in configuration. Spark allows us to override these inherited properties via
* SparkContext.setLocalProperty
*/
private void updateLocalConfiguration(final JavaSparkContext sparkContext, final Configuration configuration) {
/*
* While we could enumerate over the entire SparkConfiguration and copy into the Thread
* Local properties of the Spark Context this could cause adverse effects with future
* versions of Spark. Since the api for setting multiple local properties at once is
* restricted as private, we will only set those properties we know can effect SparkGraphComputer
* Execution rather than applying the entire configuration.
*/
final String[] validPropertyNames = {
"spark.job.description",
"spark.jobGroup.id",
"spark.job.interruptOnCancel",
"spark.scheduler.pool"
};
for (String propertyName : validPropertyNames) {
String propertyValue = configuration.get(propertyName);
if (propertyValue != null) {
this.logger.info("Setting Thread Local SparkContext Property - "
+ propertyName + " : " + propertyValue);
sparkContext.setLocalProperty(propertyName, configuration.get(propertyName));
}
}
}
内容来源于网络,如有侵权,请联系作者删除!