CODEWITHSUNDEEP
avro
magicalo
magicalo

Reputation: 473

Avro schema evolution

I have two questions:

  1. Is it possible to use the same reader and parse records that were written with two schemas that are compatible, e.g. Schema V2 only has an additional optional field compared to Schema V1 and I want the reader to understand both? I think the answer here is no, but if yes, how do I do that?

  2. I have tried writing a record with Schema V1 and reading it with Schema V2, but I get the following error:

    org.apache.avro.AvroTypeException: Found foo, expecting foo

I used avro-1.7.3 and:

   writer = new GenericDatumWriter<GenericData.Record>(SchemaV1);
   reader = new GenericDatumReader<GenericData.Record>(SchemaV2, SchemaV1);

Here are examples of the two schemas (I have tried adding a namespace as well, but no luck).

Schema V1:

{
"name": "foo",
"type": "record",
"fields": [{
    "name": "products",
    "type": {
        "type": "array",
        "items": {
            "name": "product",
            "type": "record",
            "fields": [{
                "name": "a1",
                "type": "string"
            }, {
                "name": "a2",
                "type": {"type": "fixed", "name": "a3", "size": 1}
            }, {
                "name": "a4",
                "type": "int"
            }, {
                "name": "a5",
                "type": "int"
            }]
        }
    }
}]
}

Schema V2:

{
"name": "foo",
"type": "record",
"fields": [{
    "name": "products",
    "type": {
        "type": "array",
        "items": {
            "name": "product",
            "type": "record",
            "fields": [{
                "name": "a1",
                "type": "string"
            }, {
                "name": "a2",
                "type": {"type": "fixed", "name": "a3", "size": 1}
            }, {
                "name": "a4",
                "type": "int"
            }, {
                "name": "a5",
                "type": "int"
            }]
        }
    }
},
{
            "name": "purchases",
            "type": ["null",{
                    "type": "array",
                    "items": {
                            "name": "purchase",
                            "type": "record",
                            "fields": [{
                                    "name": "a1",
                                    "type": "int"
                            }, {
                                    "name": "a2",
                                    "type": "int"
                            }]
                    }
            }]
}]
}

Thanks in advance.

Upvotes: 13

Views: 11910

Answers (2)

mahendra singh
mahendra singh

Reputation: 35

You can do opposite of it . Mean you can parse data schem 1 and write data from schema 2 . Beacause at write time it write data into file and if we don't provide any field at reading time than it will be ok. But if we write less field than read than it will not recognize extra field at reading time so , it will give error .

Upvotes: 0

Bewang
Bewang

Reputation: 483

I encountered the same issue. That might be a bug of avro, but you probably can work around by adding "default": null to the field of "purchase".

Check my blog for details: http://ben-tech.blogspot.com/2013/05/avro-schema-evolution.html

Upvotes: 12

Related Questions