org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe类的使用及代码示例

x33g5p2x  于2022-01-24 转载在 其他  
字(9.9k)|赞(0)|评价(0)|浏览(502)

本文整理了Java中org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe类的一些代码示例，展示了LazySimpleSerDe类的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。LazySimpleSerDe类的具体详情如下：
包路径：org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
类名称：LazySimpleSerDe

LazySimpleSerDe介绍

[英]LazySimpleSerDe can be used to read the same data format as MetadataTypedColumnsetSerDe and TCTLSeparatedProtocol. However, LazySimpleSerDe creates Objects in a lazy way, to provide better performance. Also LazySimpleSerDe outputs typed columns instead of treating all columns as String like MetadataTypedColumnsetSerDe.
[中]LazySimpleSerDe可用于读取与MetadataTypedColumnsetSerDe和TCTLSeparatedProtocol相同的数据格式。但是，LazySimpleSerDe以一种懒惰的方式创建对象，以提供更好的性能。此外，LazySimpleSerDe还输出类型化列，而不是将所有列视为类似字符串的MetadataTypedColumnsetSerDe。

代码示例

代码示例来源：origin: apache/hive

LazySimpleSerDe testSD = new LazySimpleSerDe();
SerDeUtils.initializeSerDe(testSD, conf, tblProps, null);
Writable s3 = testSD.serialize(s, hrsd.getObjectInspector());
LOG.info("THREE: {}", s3);
Object o3 = testSD.deserialize(s3);
Assert.assertFalse(r.getClass().equals(o3.getClass()));
HCatRecord s4 = (HCatRecord) hrsd.serialize(o3, testSD.getObjectInspector());
LOG.info("FOUR: {}", s4);
LazyHCatRecord s5 = new LazyHCatRecord(o3, testSD.getObjectInspector());
LOG.info("FIVE: {}", s5);

代码示例来源：origin: apache/hive

List<Object> list = soi.getStructFieldsDataAsList(obj);
List<? extends StructField> declaredFields = (serdeParams.getRowTypeInfo() != null && ((StructTypeInfo) serdeParams.getRowTypeInfo())
  .getAllStructFieldNames().size() > 0) ? ((StructObjectInspector) getObjectInspector())
  .getAllStructFieldRefs()
  : null;
 serializeField(serializeStream, f, foi, serdeParams);

代码示例来源：origin: klout/brickhouse

private LazySimpleSerDe getLineSerde() throws SerDeException {
  if (serde == null) {
    Logger.getLogger(LazySimpleSerDe.class).setLevel(Level.DEBUG);
    serde = new LazySimpleSerDe();
    Configuration job = new Configuration();
    Properties tbl = new Properties();
    tbl.setProperty("columns", "key,value");
    tbl.setProperty("columns.types", keyType.getTypeName() + "," + valType.getTypeName());
    serde.initialize(job, tbl);
  }
  return serde;
}

代码示例来源：origin: org.apache.hivemall/hivemall-core

StructObjectInspector lineOI = (StructObjectInspector) serde.getObjectInspector();
StructField keyRef = lineOI.getStructFieldRef("key");
StructField valueRef = lineOI.getStructFieldRef("value");
  while ((line = reader.readLine()) != null) {
    Text lineText = new Text(line);
    Object lineObj = serde.deserialize(lineText);
    List<Object> fields = lineOI.getStructFieldsDataAsList(lineObj);
    Object f0 = fields.get(0);

代码示例来源：origin: apache/hive

@Override
 public LazySimpleSerDe createSerde() throws SerializationError {
  try {
   Properties tableProps = table.getMetadata();
   tableProps.setProperty(serdeConstants.LIST_COLUMNS, Joiner.on(",").join(inputColumns));
   tableProps.setProperty(serdeConstants.LIST_COLUMN_TYPES, Joiner.on(":").join(inputTypes));
   tableProps.setProperty(serdeConstants.FIELD_DELIM, String.valueOf(fieldDelimiter));
   tableProps.setProperty(serdeConstants.COLLECTION_DELIM, String.valueOf(collectionDelimiter));
   tableProps.setProperty(serdeConstants.MAPKEY_DELIM, String.valueOf(mapKeyDelimiter));
   LazySimpleSerDe serde = new LazySimpleSerDe();
   SerDeUtils.initializeSerDe(serde, conf, tableProps, null);
   this.serde = serde;
   return serde;
  } catch (SerDeException e) {
   throw new SerializationError("Error initializing serde", e);
  }
 }
}

代码示例来源：origin: apache/hive

protected void serializeField(ByteStream.Output out, Object obj, ObjectInspector objInspector,
  LazySerDeParameters serdeParams) throws SerDeException {
 try {
  serialize(out, obj, objInspector, serdeParams.getSeparators(), 1, serdeParams.getNullSequence(),
    serdeParams.isEscaped(), serdeParams.getEscapeChar(), serdeParams.getNeedsEscape());
 } catch (IOException e) {
  throw new SerDeException(e);
 }
}

代码示例来源：origin: apache/hive

this.recordObjInspector = (LazySimpleStructObjectInspector) serde.getObjectInspector();
 this.bucketObjInspectors = getObjectInspectorsForBucketedCols(bucketIds, recordObjInspector);
} catch (SerDeException e) {

代码示例来源：origin: apache/hive

@Override
 protected void serializeField(ByteStream.Output out, Object obj, ObjectInspector objInspector,
   LazySerDeParameters serdeParams) throws SerDeException {
  if (!objInspector.getCategory().equals(Category.PRIMITIVE) || (objInspector.getTypeName().equalsIgnoreCase(serdeConstants.BINARY_TYPE_NAME))) {
   //do this for all complex types and binary
   try {
    serialize(out, SerDeUtils.getJSONString(obj, objInspector, serdeParams.getNullSequence().toString()),
      PrimitiveObjectInspectorFactory.javaStringObjectInspector, serdeParams.getSeparators(),
      1, serdeParams.getNullSequence(), serdeParams.isEscaped(), serdeParams.getEscapeChar(),
      serdeParams.getNeedsEscape());

   } catch (IOException e) {
    throw new SerDeException(e);
   }

  } else {
   //primitives except binary
   super.serializeField(out, obj, objInspector, serdeParams);
  }
 }
}

代码示例来源：origin: apache/hive

@Override
public Object encode(byte[] record) throws SerializationError {
 try {
  BytesWritable blob = new BytesWritable();
  blob.set(record, 0, record.length);
  return serde.deserialize(blob);
 } catch (SerDeException e) {
  throw new SerializationError("Unable to convert byte[] record into Object", e);
 }
}

代码示例来源：origin: com.amazon.emr/hive1-shims

Hive1SerDeParametersShim(Configuration configuration, Properties properties, String serDeName)
  throws SerDeException {
 this.realSerDeParameters = LazySimpleSerDe.initSerdeParams(configuration, properties,
   serDeName);
}

代码示例来源：origin: io.github.myui/hivemall

StructObjectInspector lineOI = (StructObjectInspector) serde.getObjectInspector();
StructField keyRef = lineOI.getStructFieldRef("key");
StructField valueRef = lineOI.getStructFieldRef("value");
  while((line = reader.readLine()) != null) {
    Text lineText = new Text(line);
    Object lineObj = serde.deserialize(lineText);
    List<Object> fields = lineOI.getStructFieldsDataAsList(lineObj);
    Object f0 = fields.get(0);

代码示例来源：origin: apache/hive

/**
 * Creates LazySimpleSerde
 * @return
 * @throws SerializationError if serde could not be initialized
 * @param tbl
 */
protected static LazySimpleSerDe createSerde(Table tbl, HiveConf conf, char serdeSeparator)
    throws SerializationError {
 try {
  Properties tableProps = MetaStoreUtils.getTableMetadata(tbl);
  tableProps.setProperty("field.delim", String.valueOf(serdeSeparator));
  LazySimpleSerDe serde = new LazySimpleSerDe();
  SerDeUtils.initializeSerDe(serde, conf, tableProps, null);
  return serde;
 } catch (SerDeException e) {
  throw new SerializationError("Error initializing serde", e);
 }
}

代码示例来源：origin: apache/hive

out.write(separator);
 serialize(out, list.get(i), eoi, separators, level + 1, nullSequence,
   escaped, escapeChar, needsEscape);
  out.write(separator);
 serialize(out, entry.getKey(), koi, separators, level + 2,
   nullSequence, escaped, escapeChar, needsEscape);
 out.write(keyValueSeparator);
 serialize(out, entry.getValue(), voi, separators, level + 2,
   nullSequence, escaped, escapeChar, needsEscape);
  out.write(separator);
 serialize(out, list.get(i), fields.get(i).getFieldObjectInspector(),
   separators, level + 1, nullSequence, escaped, escapeChar,
   needsEscape);
  escaped, escapeChar, needsEscape);
out.write(separator);
serialize(out, uoi.getField(obj), ois.get(uoi.getTag(obj)),
  separators, level + 1, nullSequence, escaped, escapeChar,
  needsEscape);

代码示例来源：origin: org.apache.hive/hive-serde

@Override
 protected void serializeField(ByteStream.Output out, Object obj, ObjectInspector objInspector,
   LazySerDeParameters serdeParams) throws SerDeException {
  if (!objInspector.getCategory().equals(Category.PRIMITIVE) || (objInspector.getTypeName().equalsIgnoreCase(serdeConstants.BINARY_TYPE_NAME))) {
   //do this for all complex types and binary
   try {
    serialize(out, SerDeUtils.getJSONString(obj, objInspector, serdeParams.getNullSequence().toString()),
      PrimitiveObjectInspectorFactory.javaStringObjectInspector, serdeParams.getSeparators(),
      1, serdeParams.getNullSequence(), serdeParams.isEscaped(), serdeParams.getEscapeChar(),
      serdeParams.getNeedsEscape());

   } catch (IOException e) {
    throw new SerDeException(e);
   }

  } else {
   //primitives except binary
   super.serializeField(out, obj, objInspector, serdeParams);
  }
 }
}

代码示例来源：origin: apache/hive

@Override
public Object encode(byte[] record) throws SerializationError {
 try {
  BytesWritable blob = new BytesWritable();
  blob.set(record, 0, record.length);
  return serde.deserialize(blob);
 } catch (SerDeException e) {
  throw new SerializationError("Unable to convert byte[] record into Object", e);
 }
}

代码示例来源：origin: awslabs/emr-dynamodb-connector

Hive1SerDeParametersShim(Configuration configuration, Properties properties, String serDeName)
  throws SerDeException {
 this.realSerDeParameters = LazySimpleSerDe.initSerdeParams(configuration, properties,
   serDeName);
}

代码示例来源：origin: io.github.myui/hivemall-core

StructObjectInspector lineOI = (StructObjectInspector) serde.getObjectInspector();
StructField keyRef = lineOI.getStructFieldRef("key");
StructField valueRef = lineOI.getStructFieldRef("value");
  while ((line = reader.readLine()) != null) {
    Text lineText = new Text(line);
    Object lineObj = serde.deserialize(lineText);
    List<Object> fields = lineOI.getStructFieldsDataAsList(lineObj);
    Object f0 = fields.get(0);

代码示例来源：origin: io.github.myui/hivemall

public static LazySimpleSerDe getKeyValueLineSerde(@Nonnull final PrimitiveObjectInspector keyOI, @Nonnull final PrimitiveObjectInspector valueOI)
    throws SerDeException {
  LazySimpleSerDe serde = new LazySimpleSerDe();
  Configuration conf = new Configuration();
  Properties tbl = new Properties();
  tbl.setProperty("columns", "key,value");
  tbl.setProperty("columns.types", keyOI.getTypeName() + "," + valueOI.getTypeName());
  serde.initialize(conf, tbl);
  return serde;
}

代码示例来源：origin: apache/hive

String types = typesSb.toString();
serde = new LazySimpleSerDe();
Properties props = new Properties();
if (names.length() > 0) {

代码示例来源：origin: org.apache.hadoop.hive/hive-serde

List<Object> list = soi.getStructFieldsDataAsList(obj);
List<? extends StructField> declaredFields = (serdeParams.rowTypeInfo != null && ((StructTypeInfo) serdeParams.rowTypeInfo)
  .getAllStructFieldNames().size() > 0) ? ((StructObjectInspector) getObjectInspector())
  .getAllStructFieldRefs()
  : null;
 serializeField(serializeStream, f, foi, serdeParams);

内容来源于网络，如有侵权，请联系作者删除！

LazySimpleSerDe

相关文章

    查看更多
    微信公众号

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com