使用Avro Schema和JSON数据创建Avro GenericRecord

ig9co6j1  于 5个月前  发布在  其他
关注(0)|答案(1)|浏览(56)

我尝试使用以下模式和JSON数据生成Avro GenericRecord。
Avro架构

{
  "type": "record",
  "name": "Person",
  "fields": [
    {
      "name": "name",
      "type": "string"
    },
    {
      "name": "age",
      "type": "int"
    },
    {
      "name": "city",
      "type": "string"
    },
    {
      "name": "gender",
      "type": {
        "type": "enum",
        "name": "Gender",
        "symbols": ["MALE", "FEMALE"]
      }
    }
  ]
}

字符串
JSON数据

{"name": "John", "age": 30, "city": "New York", "gender": "MALE"}


我用来生成GenericRecord的代码是

public class Main {

    public static void main(String[] args) throws IOException {
        Schema schema = readSchema();
        JsonNode data = readData();
        GenericRecord genericRecord = convertJsonToAvro(data, schema);
        System.out.println(genericRecord);
    }

    public static GenericRecord convertJsonToAvro(JsonNode jsonNode, Schema avroSchema) throws IOException {
        DatumReader<GenericRecord> reader = new GenericDatumReader<>(avroSchema);
        Decoder decoder = DecoderFactory.get().jsonDecoder(avroSchema, jsonNode.toString());
        return reader.read(null, decoder);
    }

    private static Schema readSchema() throws IOException {
        InputStream inputStream = Main.class.getClassLoader().getResourceAsStream("schemas/person.avsc");
        return new Schema.Parser().parse(inputStream);

    }

    private static JsonNode readData() throws IOException {
        InputStream inputStream = Main.class.getClassLoader().getResourceAsStream("sample_data/person.json");
        ObjectMapper objectMapper = new ObjectMapper();
        return objectMapper.readValue(inputStream, JsonNode.class);
    }

}


从上面的代码中,我可以成功地生成一个GenericRecord。但是当schemas enum字段被更改为下面的可选字段时,它会抛出一个错误。
可选字段

{
    "name": "gender",
    "type": [
        "null",
        {
            "type": "enum",
            "name": "Gender",
            "symbols": ["MALE", "FEMALE"]
        }
    ],
    "default": null
}


误差

Exception in thread "main" org.apache.avro.AvroTypeException: Expected start-union. Got VALUE_STRING
at org.apache.avro.io.JsonDecoder.error(JsonDecoder.java:511)
at org.apache.avro.io.JsonDecoder.readIndex(JsonDecoder.java:430)
at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:282)
at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:188)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:161)
at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:260)
at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:248)
at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:180)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:161)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:154)
at com.generic.Main.convertJsonToAvro(Main.java:27)
at com.generic.Main.main(Main.java:20)

FAILURE: Build failed with an exception.


我非常感谢如果有人能解释这里发生了什么,以及如何修复这个错误.我知道关于allegro/json-avro-converter,但我想这样做没有它.一个代码示例将是非常有帮助的.
请找到我在上面的代码中使用的依赖项

implementation group: 'org.apache.avro', name: 'avro', version: '1.11.3'
implementation group: 'com.fasterxml.jackson.core', name: 'jackson-core', version: '2.16.0'

li9yvcax

li9yvcax1#

已关闭

gender端的枚举类型是Gender,当您将该属性视为对象时,负载应将其作为子属性包含。
这是一个非空的gender负载:

{"name": "John", "age": 30, "city": "New York", "gender": {"Gender": "MALE"}}

字符串
下面是GenericRecord

{"name": "John", "age": 30, "city": "New York", "gender": "MALE"}


这是空gender负载

{"name": "John", "age": 30, "city": "New York", "gender": null}

相关问题