CODEWITHSUNDEEP
apache-flinkavro
Jeenson Ephraim
Jeenson Ephraim

Reputation: 571

JSON to Avro Decoding - AvroTypeException: Expected field name not found

I am using Avro 1.8.2 and I am trying to convert JSON into a GenericRecord

 DatumReader<GenericData.Record> datumReader = new GenericDatumReader<>(schema);
 Decoder decoder = DecoderFactory.get().jsonDecoder(schema, jsonStr);
 datumReader.read(null, decoder)

I get JSON data from a third party and I have no control over the elements. The AVRO schema is

{
    "namespace":"com.avro.generated",
    "type":"record",
    "name":"TestEvent",
    "fields":[
        {"name":"userId","type":"string"},
        {"name":"frm","type":"string"},
        {"name":"issuerName","type":"string"},
        {"name":"profileId","type":"string"}
    ]
}

If I use this JSON

{
    "userId":"5435tert34tgcb21391f7bda71",
    "frm":"somerm",
    "issuerName":"somenameorts",
    "profileId":"0werwerwer0000-0000-000000000000"
}

It works fine. However if the json does not contain the frm element as shown below

{
    "userId":"5435tert34tgcb21391f7bda71",
    "issuerName":"somenameorts",
    "profileId":"0werwerwer0000-0000-000000000000"
}

Then I get this exception

org.apache.avro.AvroTypeException: Expected field name not found: frm.

Is there any way to make this work?. I have no control over the JSON. I have read other SO posts about using schemas like

{"name":"frm","type":["null","string"],"default": "null"}

But none of this is working

Thanks

Upvotes: 3

Views: 4414

Answers (2)

Jeenson Ephraim
Jeenson Ephraim

Reputation: 571

The only way I am able to make this work is by converting it into the actual generated class which extends SpecificRecordBase and the converting it back into a GenericRecord. I dont know if this is a valid way of doing it , but just posting it

ObjectMapper mapper = new ObjectMapper();
TestEvent node = mapper.readValue(jsonStr.getBytes(), TestEvent.class);
// This gives the actual Class

I can convert this back to a generic record using

Schema schema = TestEvent.getClassSchema();
ReflectDatumWriter datumWriter = new ReflectDatumWriter<>(schema);
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();

BinaryEncoder encoder = EncoderFactory.get().binaryEncoder(outputStream, null);
datumWriter.write(node, encoder);
encoder.flush();

DatumReader<GenericRecord> datumReader = new GenericDatumReader<>(schema);
BinaryDecoder  decoder = DecoderFactory.get().binaryDecoder(outputStream.toByteArray(), null);
datumReader.read(null, decoder)

Upvotes: 0

BenoitParis
BenoitParis

Reputation: 3184

All fields are mandatory in AVRO, but you can provide a default so that it has the field.

{
    "namespace":"com.avro.generated",
    "type":"record",
    "name":"TestEvent",
    "fields":[
        {"name":"userId","type":["null","string"], "default": null},
        {"name":"frm","type":["null","string"], "default": null},
        {"name":"issuerName","type":["null","string"], "default": null},
        {"name":"profileId","type":["null","string"], "default": null}
    ]
}

EDIT: sorry didn't read the end of your message. What is the error when you say it is not working (also, notice that null is not to be quoted)

Upvotes: 2

Related Questions