com.alibaba.datax.common.util.Configuration.getConfiguration()方法的使用及代码示例

x33g5p2x  于2022-01-18 转载在 其他  
字(10.1k)|赞(0)|评价(0)|浏览(121)

本文整理了Java中com.alibaba.datax.common.util.Configuration.getConfiguration()方法的一些代码示例，展示了Configuration.getConfiguration()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。Configuration.getConfiguration()方法的具体详情如下：
包路径：com.alibaba.datax.common.util.Configuration
类名称：Configuration
方法名：getConfiguration

Configuration.getConfiguration介绍

[英]用户指定部分path，获取Configuration的子集

如果path获取的路径或者对象不存在，返回null
[中]用户指定部分路径获取配置的子集
如果路径获取的路径或者对象不存在，返回无效的

代码示例

代码示例来源：origin: ECNU-1X/DataX-Masking

private static Configuration getPluginConf(PluginType pluginType,
                      String pluginName) {
  Configuration pluginConf = pluginRegisterCenter
      .getConfiguration(generatePluginKey(pluginType, pluginName));
  if (null == pluginConf) {
    throw DataXException.asDataXException(
        FrameworkErrorCode.PLUGIN_INSTALL_ERROR,
        String.format("DataX不能找到插件[%s]的配置.",
            pluginName));
  }
  return pluginConf;
}

代码示例来源：origin: ECNU-1X/DataX-Masking

public static String filterJobConfiguration(final Configuration configuration) {
  Configuration jobConfWithSetting = configuration.getConfiguration("job").clone();
  Configuration jobContent = jobConfWithSetting.getConfiguration("content");
  filterSensitiveConfiguration(jobContent);
  jobConfWithSetting.set("content",jobContent);
  return jobConfWithSetting.beautify();
}

代码示例来源：origin: ECNU-1X/DataX-Masking

public DFSUtil(Configuration taskConfig) {
  hadoopConf = new org.apache.hadoop.conf.Configuration();
  //io.file.buffer.size 性能参数
  //http://blog.csdn.net/yangjl38/article/details/7583374
  Configuration hadoopSiteParams = taskConfig.getConfiguration(Key.HADOOP_CONFIG);
  JSONObject hadoopSiteParamsAsJsonObject = JSON.parseObject(taskConfig.getString(Key.HADOOP_CONFIG));
  if (null != hadoopSiteParams) {
    Set<String> paramKeys = hadoopSiteParams.getKeys();
    for (String each : paramKeys) {
      hadoopConf.set(each, hadoopSiteParamsAsJsonObject.getString(each));
    }
  }
  hadoopConf.set(HDFS_DEFAULTFS_KEY, taskConfig.getString(Key.DEFAULT_FS));
  //是否有Kerberos认证
  this.haveKerberos = taskConfig.getBool(Key.HAVE_KERBEROS, false);
  if (haveKerberos) {
    this.kerberosKeytabFilePath = taskConfig.getString(Key.KERBEROS_KEYTAB_FILE_PATH);
    this.kerberosPrincipal = taskConfig.getString(Key.KERBEROS_PRINCIPAL);
    this.hadoopConf.set(HADOOP_SECURITY_AUTHENTICATION_KEY, "kerberos");
  }
  this.kerberosAuthentication(this.kerberosPrincipal, this.kerberosKeytabFilePath);
  LOG.info(String.format("hadoopConfig details:%s", JSON.toJSONString(this.hadoopConf)));
}

代码示例来源：origin: ECNU-1X/DataX-Masking

public void getFileSystem(String defaultFS, Configuration taskConfig){
  hadoopConf = new org.apache.hadoop.conf.Configuration();
  Configuration hadoopSiteParams = taskConfig.getConfiguration(Key.HADOOP_CONFIG);
  JSONObject hadoopSiteParamsAsJsonObject = JSON.parseObject(taskConfig.getString(Key.HADOOP_CONFIG));
  if (null != hadoopSiteParams) {

代码示例来源：origin: ECNU-1X/DataX-Masking

public TaskExecutor(Configuration taskConf, int attemptCount) {
  Validate.isTrue(null != this.taskConfig.getConfiguration(CoreConstant.JOB_READER)
          && null != this.taskConfig.getConfiguration(CoreConstant.JOB_WRITER),
      "[reader|writer]的插件参数不能为空!");

代码示例来源：origin: ECNU-1X/DataX-Masking

private static void validateVersionColumn(com.alibaba.datax.common.util.Configuration originalConfig){
    Configuration versionColumn = originalConfig.getConfiguration(Key.VERSION_COLUMN);
    //为null,表示用当前时间;指定列,需要index
    if(versionColumn != null){
      Integer index = versionColumn.getInt(Key.INDEX);
      if(index == null ){
        throw DataXException.asDataXException(Hbase11xWriterErrorCode.REQUIRED_VALUE, "versionColumn配置项中index为必填项");
      }
      if(index == -1){
        //指定时间,需要index=-1,value
        versionColumn.getNecessaryValue(Key.VALUE,Hbase11xWriterErrorCode.REQUIRED_VALUE);
      }else if(index < 0){
        throw DataXException.asDataXException(Hbase11xWriterErrorCode.ILLEGAL_VALUE, "您versionColumn配置项中index配置不正确,只能取-1或者非负数");
      }
    }
  }
}

代码示例来源：origin: ECNU-1X/DataX-Masking

public static void validateVersionColumn(com.alibaba.datax.common.util.Configuration originalConfig){
    Configuration versionColumn = originalConfig.getConfiguration(Key.VERSION_COLUMN);
    //为null,表示用当前时间;指定列,需要index
    if(versionColumn != null){
      Integer index = versionColumn.getInt(Key.INDEX);
      if(index == null ){
        throw DataXException.asDataXException(Hbase094xWriterErrorCode.REQUIRED_VALUE, "versionColumn配置项中index为必填项");
      }
      if(index == -1){
        //指定时间,需要index=-1,value
        versionColumn.getNecessaryValue(Key.VALUE, Hbase094xWriterErrorCode.REQUIRED_VALUE);
      }else if(index < 0){
        throw DataXException.asDataXException(Hbase094xWriterErrorCode.ILLEGAL_VALUE, "您versionColumn配置项中index配置不正确,只能取-1或者非负数");
      }
    }
  }
}

代码示例来源：origin: ECNU-1X/DataX-Masking

private Reader.Job preCheckReaderInit(JobPluginCollector jobPluginCollector) {
  this.readerPluginName = this.configuration.getString(
      CoreConstant.DATAX_JOB_CONTENT_READER_NAME);
  classLoaderSwapper.setCurrentThreadClassLoader(LoadUtil.getJarLoader(
      PluginType.READER, this.readerPluginName));
  Reader.Job jobReader = (Reader.Job) LoadUtil.loadJobPlugin(
      PluginType.READER, this.readerPluginName);
  this.configuration.set(CoreConstant.DATAX_JOB_CONTENT_READER_PARAMETER + ".dryRun", true);
  // 设置reader的jobConfig
  jobReader.setPluginJobConf(this.configuration.getConfiguration(
      CoreConstant.DATAX_JOB_CONTENT_READER_PARAMETER));
  // 设置reader的readerConfig
  jobReader.setPeerPluginJobConf(this.configuration.getConfiguration(
      CoreConstant.DATAX_JOB_CONTENT_READER_PARAMETER));
  jobReader.setJobPluginCollector(jobPluginCollector);
  classLoaderSwapper.restoreCurrentThreadClassLoader();
  return jobReader;
}

代码示例来源：origin: ECNU-1X/DataX-Masking

private Writer.Job preCheckWriterInit(JobPluginCollector jobPluginCollector) {
  this.writerPluginName = this.configuration.getString(
      CoreConstant.DATAX_JOB_CONTENT_WRITER_NAME);
  classLoaderSwapper.setCurrentThreadClassLoader(LoadUtil.getJarLoader(
      PluginType.WRITER, this.writerPluginName));
  Writer.Job jobWriter = (Writer.Job) LoadUtil.loadJobPlugin(
      PluginType.WRITER, this.writerPluginName);
  this.configuration.set(CoreConstant.DATAX_JOB_CONTENT_WRITER_PARAMETER + ".dryRun", true);
  // 设置writer的jobConfig
  jobWriter.setPluginJobConf(this.configuration.getConfiguration(
      CoreConstant.DATAX_JOB_CONTENT_WRITER_PARAMETER));
  // 设置reader的readerConfig
  jobWriter.setPeerPluginJobConf(this.configuration.getConfiguration(
      CoreConstant.DATAX_JOB_CONTENT_READER_PARAMETER));
  jobWriter.setPeerPluginName(this.readerPluginName);
  jobWriter.setJobPluginCollector(jobPluginCollector);
  classLoaderSwapper.restoreCurrentThreadClassLoader();
  return jobWriter;
}

代码示例来源：origin: ECNU-1X/DataX-Masking

/**
 * reader job的初始化，返回Reader.Job
 *
 * @return
 */
private Reader.Job initJobReader(
    JobPluginCollector jobPluginCollector) {
  this.readerPluginName = this.configuration.getString(
      CoreConstant.DATAX_JOB_CONTENT_READER_NAME);
  classLoaderSwapper.setCurrentThreadClassLoader(LoadUtil.getJarLoader(
      PluginType.READER, this.readerPluginName));
  Reader.Job jobReader = (Reader.Job) LoadUtil.loadJobPlugin(
      PluginType.READER, this.readerPluginName);
  // 设置reader的jobConfig
  jobReader.setPluginJobConf(this.configuration.getConfiguration(
      CoreConstant.DATAX_JOB_CONTENT_READER_PARAMETER));
  // 设置reader的readerConfig
  jobReader.setPeerPluginJobConf(this.configuration.getConfiguration(
      CoreConstant.DATAX_JOB_CONTENT_WRITER_PARAMETER));
  jobReader.setJobPluginCollector(jobPluginCollector);
  jobReader.init();
  classLoaderSwapper.restoreCurrentThreadClassLoader();
  return jobReader;
}

代码示例来源：origin: ECNU-1X/DataX-Masking

LOG.info(String.format(" %s of transformer init success. name=%s, isNative=%s parameter = %s"
    , i, transformerInfo.getTransformer().getTransformerName()
    , transformerInfo.isNative(), configuration.getConfiguration("parameter")));

代码示例来源：origin: ECNU-1X/DataX-Masking

public HbaseAbstractTask(com.alibaba.datax.common.util.Configuration configuration) {
  this.htable = Hbase094xHelper.getTable(configuration);
  this.columns = configuration.getListConfiguration(Key.COLUMN);
  this.rowkeyColumn = configuration.getListConfiguration(Key.ROWKEY_COLUMN);
  this.versionColumn = configuration.getConfiguration(Key.VERSION_COLUMN);
  this.encoding = configuration.getString(Key.ENCODING,Constant.DEFAULT_ENCODING);
  this.nullMode = NullModeType.getByTypeName(configuration.getString(Key.NULL_MODE,Constant.DEFAULT_NULL_MODE));
  this.walFlag = configuration.getBool(Key.WAL_FLAG, false);
}

代码示例来源：origin: ECNU-1X/DataX-Masking

public HbaseAbstractTask(com.alibaba.datax.common.util.Configuration configuration) {
  //this.htable = Hbase11xHelper.getTable(configuration);
  this.bufferedMutator = Hbase11xHelper.getBufferedMutator(configuration);
  this.columns = configuration.getListConfiguration(Key.COLUMN);
  this.rowkeyColumn = configuration.getListConfiguration(Key.ROWKEY_COLUMN);
  this.versionColumn = configuration.getConfiguration(Key.VERSION_COLUMN);
  this.encoding = configuration.getString(Key.ENCODING,Constant.DEFAULT_ENCODING);
  this.nullMode = NullModeType.getByTypeName(configuration.getString(Key.NULL_MODE,Constant.DEFAULT_NULL_MODE));
  this.walFlag = configuration.getBool(Key.WAL_FLAG, false);
}

代码示例来源：origin: ECNU-1X/DataX-Masking

/**
 * writer job的初始化，返回Writer.Job
 *
 * @return
 */
private Writer.Job initJobWriter(
    JobPluginCollector jobPluginCollector) {
  this.writerPluginName = this.configuration.getString(
      CoreConstant.DATAX_JOB_CONTENT_WRITER_NAME);
  classLoaderSwapper.setCurrentThreadClassLoader(LoadUtil.getJarLoader(
      PluginType.WRITER, this.writerPluginName));
  Writer.Job jobWriter = (Writer.Job) LoadUtil.loadJobPlugin(
      PluginType.WRITER, this.writerPluginName);
  // 设置writer的jobConfig
  jobWriter.setPluginJobConf(this.configuration.getConfiguration(
      CoreConstant.DATAX_JOB_CONTENT_WRITER_PARAMETER));
  // 设置reader的readerConfig
  jobWriter.setPeerPluginJobConf(this.configuration.getConfiguration(
      CoreConstant.DATAX_JOB_CONTENT_READER_PARAMETER));
  jobWriter.setPeerPluginName(this.readerPluginName);
  jobWriter.setJobPluginCollector(jobPluginCollector);
  jobWriter.init();
  classLoaderSwapper.restoreCurrentThreadClassLoader();
  return jobWriter;
}

代码示例来源：origin: ECNU-1X/DataX-Masking

Configuration jobInfoConfig = allConf.getConfiguration(CoreConstant.DATAX_JOB_JOBINFO);

代码示例来源：origin: ECNU-1X/DataX-Masking

newRunner = LoadUtil.loadPluginRunner(pluginType,
    this.taskConfig.getString(CoreConstant.JOB_READER_NAME));
newRunner.setJobConf(this.taskConfig.getConfiguration(
    CoreConstant.JOB_READER_PARAMETER));
    this.taskConfig.getString(CoreConstant.JOB_WRITER_NAME));
newRunner.setJobConf(this.taskConfig
    .getConfiguration(CoreConstant.JOB_WRITER_PARAMETER));

内容来源于网络，如有侵权，请联系作者删除！

Configuration

相关文章

    查看更多
    微信公众号

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com