com.alibaba.datax.common.util.Configuration.getList()方法的使用及代码示例

x33g5p2x  于2022-01-18 转载在 其他  
字(10.0k)|赞(0)|评价(0)|浏览(183)

本文整理了Java中com.alibaba.datax.common.util.Configuration.getList()方法的一些代码示例，展示了Configuration.getList()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。Configuration.getList()方法的具体详情如下：
包路径：com.alibaba.datax.common.util.Configuration
类名称：Configuration
方法名：getList

Configuration.getList介绍

[英]根据用户提供的json path，寻址List对象，如果对象不存在，返回null
[中]根据用户提供的json路径寻址列表对象，如果对象不存在，返回无效的

代码示例

代码示例来源：origin: ECNU-1X/DataX-Masking

/**
 * 根据用户提供的json path，寻址List对象，如果对象不存在，返回默认List
 */
@SuppressWarnings("unchecked")
public List<Object> getList(final String path,
    final List<Object> defaultList) {
  Object object = this.getList(path);
  if (null == object) {
    return defaultList;
  }
  return (List<Object>) object;
}

代码示例来源：origin: ECNU-1X/DataX-Masking

/**
 * 根据用户提供的json path，寻址List对象，如果对象不存在，返回默认List
 */
public <T> List<T> getList(final String path, final List<T> defaultList,
    Class<T> t) {
  List<T> list = this.getList(path, t);
  if (null == list) {
    return defaultList;
  }
  return list;
}

代码示例来源：origin: ECNU-1X/DataX-Masking

public MultiVersionDynamicColumnTask(Configuration configuration){
  super(configuration);
  this.columnFamilies = configuration.getList(Key.COLUMN_FAMILY, String.class);
}

代码示例来源：origin: ECNU-1X/DataX-Masking

public static List<ColumnEntry> getListColumnEntry(
    Configuration configuration, final String path) {
  List<JSONObject> lists = configuration.getList(path, JSONObject.class);
  if (lists == null) {
    return null;
  }
  List<ColumnEntry> result = new ArrayList<ColumnEntry>();
  for (final JSONObject object : lists) {
    result.add(JSON.parseObject(object.toJSONString(),
        ColumnEntry.class));
  }
  return result;
}

代码示例来源：origin: ECNU-1X/DataX-Masking

public NormalTask(Configuration configuration) {
  super(configuration);
  this.column = configuration.getList(Key.COLUMN, Map.class);
  this.hbaseColumnCells = Hbase094xHelper.parseColumnOfNormalMode(this.column);
}

代码示例来源：origin: ECNU-1X/DataX-Masking

@Override
public void init() {
  this.readerSliceConfig = this.getPluginJobConf();
  this.sourceFiles = this.readerSliceConfig.getList(
      Constant.SOURCE_FILES, String.class);
}

代码示例来源：origin: ECNU-1X/DataX-Masking

public NormalTask(Configuration configuration) {
  super(configuration);
  this.column = configuration.getList(Key.COLUMN, Map.class);
  this.hbaseColumnCells = Hbase11xHelper.parseColumnOfNormalMode(this.column);
}

代码示例来源：origin: ECNU-1X/DataX-Masking

/**
 * 根据用户提供的json path，寻址包含Configuration的List，如果对象不存在，返回默认null
 */
public List<Configuration> getListConfiguration(final String path) {
  List<Object> lists = getList(path);
  if (lists == null) {
    return null;
  }
  List<Configuration> result = new ArrayList<Configuration>();
  for (final Object object : lists) {
    result.add(Configuration.from(Configuration.toJSONString(object)));
  }
  return result;
}

代码示例来源：origin: ECNU-1X/DataX-Masking

public static void checkNecessaryConfig(Configuration originalConfig) {
  originalConfig.getNecessaryValue(Key.ODPS_SERVER, OdpsReaderErrorCode.REQUIRED_VALUE);
  originalConfig.getNecessaryValue(Key.PROJECT, OdpsReaderErrorCode.REQUIRED_VALUE);
  originalConfig.getNecessaryValue(Key.TABLE, OdpsReaderErrorCode.REQUIRED_VALUE);
  if (null == originalConfig.getList(Key.COLUMN) ||
      originalConfig.getList(Key.COLUMN, String.class).isEmpty()) {
    throw DataXException.asDataXException(OdpsReaderErrorCode.REQUIRED_VALUE, "datax获取不到源表的列信息， 由于您未配置读取源头表的列信息. datax无法知道该抽取表的哪些字段的数据 " +
        "正确的配置方式是给 column 配置上您需要读取的列名称,用英文逗号分隔.");
  }
}

代码示例来源：origin: ECNU-1X/DataX-Masking

public MultiVersionTask(Configuration configuration) {
  super(configuration);
  this.maxVersion = configuration.getInt(Key.MAX_VERSION);
  this.column = configuration.getList(Key.COLUMN, Map.class);
  this.familyQualifierMap = Hbase11xHelper.parseColumnOfMultiversionMode(this.column);
  try {
    MultiVersionTask.COLON_BYTE = ":".getBytes("utf8");
  } catch (UnsupportedEncodingException e) {
    throw DataXException.asDataXException(Hbase11xReaderErrorCode.PREPAR_READ_ERROR, "系统内部获取 列族与列名冒号分隔符的二进制时失败.", e);
  }
}

代码示例来源：origin: ECNU-1X/DataX-Masking

public MultiVersionTask(Configuration configuration) {
  super(configuration);
  this.maxVersion = configuration.getInt(Key.MAX_VERSION);
  this.column = configuration.getList(Key.COLUMN, Map.class);
  this.familyQualifierMap = Hbase094xHelper.parseColumnOfMultiversionMode(this.column);
  try {
    MultiVersionTask.COLON_BYTE = ":".getBytes("utf8");
  } catch (UnsupportedEncodingException e) {
    throw DataXException.asDataXException(Hbase094xReaderErrorCode.PREPAR_READ_ERROR, "系统内部获取 列族与列名冒号分隔符的二进制时失败.", e);
  }
}

代码示例来源：origin: ECNU-1X/DataX-Masking

public static List<Object> checkListAndGet(Configuration param, String key, boolean isCheckEmpty) {
  List<Object> value = null;
  try {
    value = param.getList(key);
  } catch (ClassCastException e) {
    throwNotListException(key);
  }
  if (null == value) {
    throwNotExistException(key);
  } else if (isCheckEmpty && value.isEmpty()) {
    throwEmptyException(key);
  }
  return value;
}

代码示例来源：origin: ECNU-1X/DataX-Masking

public static List<Object> checkListAndGet(Configuration param, String key, boolean isCheckEmpty) {
  List<Object> value = null;
  try {
    value = param.getList(key);
  } catch (ClassCastException e) {
    throwNotListException(key);
  }
  if (null == value) {
    throwNotExistException(key);
  } else if (isCheckEmpty && value.isEmpty()) {
    throwEmptyListException(key);
  }
  return value;
}

代码示例来源：origin: ECNU-1X/DataX-Masking

private void dealColumn(Configuration originalConfig, List<String> allColumns) {
  //之前已经检查了userConfiguredColumns 一定不为空
  List<String> userConfiguredColumns = originalConfig.getList(Key.COLUMN, String.class);
  if (1 == userConfiguredColumns.size() && "*".equals(userConfiguredColumns.get(0))) {
    userConfiguredColumns = allColumns;
    originalConfig.set(Key.COLUMN, allColumns);
  } else {
    //检查列是否重复，大小写不敏感（所有写入，都是不允许写入段的列重复的）
    ListUtil.makeSureNoValueDuplicate(userConfiguredColumns, false);
    //检查列是否存在，大小写不敏感
    ListUtil.makeSureBInA(allColumns, userConfiguredColumns, false);
  }
  List<Integer> columnPositions = OdpsUtil.parsePosition(allColumns, userConfiguredColumns);
  originalConfig.set(Constant.COLUMN_POSITION, columnPositions);
}

代码示例来源：origin: ECNU-1X/DataX-Masking

@Override
public void init() {
  this.readerSliceConfig = super.getPluginJobConf();
  this.columns = this.readerSliceConfig.getList(Key.COLUMN,
      String.class);
  this.sliceRecordCount = this.readerSliceConfig
      .getLong(Key.SLICE_RECORD_COUNT);
  this.haveMixupFunction = this.readerSliceConfig.getBool(
      Constant.HAVE_MIXUP_FUNCTION, false);
}

代码示例来源：origin: ECNU-1X/DataX-Masking

public static MongoClient initMongoClient(Configuration conf) {
  List<Object> addressList = conf.getList(KeyConstant.MONGO_ADDRESS);
  if(addressList == null || addressList.size() <= 0) {
    throw DataXException.asDataXException(MongoDBReaderErrorCode.ILLEGAL_VALUE,"不合法参数");
  }
  try {
    return new MongoClient(parseServerAddress(addressList));
  } catch (UnknownHostException e) {
    throw DataXException.asDataXException(MongoDBReaderErrorCode.ILLEGAL_ADDRESS,"不合法的地址");
  } catch (NumberFormatException e) {
    throw DataXException.asDataXException(MongoDBReaderErrorCode.ILLEGAL_VALUE,"不合法参数");
  } catch (Exception e) {
    throw DataXException.asDataXException(MongoDBReaderErrorCode.UNEXCEPT_EXCEPTION,"未知异常");
  }
}

代码示例来源：origin: ECNU-1X/DataX-Masking

public static MongoClient initMongoClient(Configuration conf) {
  List<Object> addressList = conf.getList(KeyConstant.MONGO_ADDRESS);
  if(addressList == null || addressList.size() <= 0) {
    throw DataXException.asDataXException(MongoDBWriterErrorCode.ILLEGAL_VALUE,"不合法参数");
  }
  try {
    return new MongoClient(parseServerAddress(addressList));
  } catch (UnknownHostException e) {
    throw DataXException.asDataXException(MongoDBWriterErrorCode.ILLEGAL_ADDRESS,"不合法的地址");
  } catch (NumberFormatException e) {
    throw DataXException.asDataXException(MongoDBWriterErrorCode.ILLEGAL_VALUE,"不合法参数");
  } catch (Exception e) {
    throw DataXException.asDataXException(MongoDBWriterErrorCode.UNEXCEPT_EXCEPTION,"未知异常");
  }
}

代码示例来源：origin: ECNU-1X/DataX-Masking

public static MongoClient initCredentialMongoClient(Configuration conf, String userName, String password, String database) {
  List<Object> addressList = conf.getList(KeyConstant.MONGO_ADDRESS);
  if(!isHostPortPattern(addressList)) {
    throw DataXException.asDataXException(MongoDBReaderErrorCode.ILLEGAL_VALUE,"不合法参数");
  }
  try {
    MongoCredential credential = MongoCredential.createCredential(userName, database, password.toCharArray());
    return new MongoClient(parseServerAddress(addressList), Arrays.asList(credential));
  } catch (UnknownHostException e) {
    throw DataXException.asDataXException(MongoDBReaderErrorCode.ILLEGAL_ADDRESS,"不合法的地址");
  } catch (NumberFormatException e) {
    throw DataXException.asDataXException(MongoDBReaderErrorCode.ILLEGAL_VALUE,"不合法参数");
  } catch (Exception e) {
    throw DataXException.asDataXException(MongoDBReaderErrorCode.UNEXCEPT_EXCEPTION,"未知异常");
  }
}
/**

代码示例来源：origin: ECNU-1X/DataX-Masking

@Override
public void init() {
  this.taskConfig = super.getPluginJobConf();
  this.sourceFiles = this.taskConfig.getList(Constant.SOURCE_FILES, String.class);
  this.specifiedFileType = this.taskConfig.getNecessaryValue(Key.FILETYPE, HdfsReaderErrorCode.REQUIRED_VALUE);
  this.encoding = this.taskConfig.getString(com.alibaba.datax.plugin.unstructuredstorage.reader.Key.ENCODING, "UTF-8");
  this.dfsUtil = new DFSUtil(this.taskConfig);
  this.bufferSize = this.taskConfig.getInt(com.alibaba.datax.plugin.unstructuredstorage.reader.Key.BUFFER_SIZE,
      com.alibaba.datax.plugin.unstructuredstorage.reader.Constant.DEFAULT_BUFFER_SIZE);
}

代码示例来源：origin: ECNU-1X/DataX-Masking

public void startWrite(RecordReceiver recordReceiver) {
  // 这里的是非odps数据源->odps中转临时表数据同步, load操作在job post阶段完成
  if(Constant.LOADMODE.equalsIgnoreCase(this.writeMode)) {
    odpsWriterTaskProxy.setTaskPluginCollector(super.getTaskPluginCollector());
    odpsWriterTaskProxy.startWrite(recordReceiver);
  } else {
    // insert 模式
    List<String> columns = writerSliceConfig.getList(Key.COLUMN, String.class);
    Connection connection = AdsUtil.getAdsConnect(this.writerSliceConfig);
    TaskPluginCollector taskPluginCollector = super.getTaskPluginCollector();
    AdsInsertProxy proxy = new AdsInsertProxy(schema + "." + table, columns, writerSliceConfig, taskPluginCollector, this.tableInfo);
    proxy.startWriteWithConnection(recordReceiver, connection, columnNumber);
  }
}

内容来源于网络，如有侵权，请联系作者删除！

Configuration

相关文章

    查看更多
    微信公众号

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com