CODEWITHSUNDEEP
javaserializationavrojsonschema
Richard Le
Richard Le

Reputation: 669

Generate Avro Schema from certain Java Object

Apache Avro provides a compact, fast, binary data format, rich data structure for serialization. However, it requires user to define a schema (in JSON) for object which need to be serialized.

In some case, this can not be possible (e.g: the class of that Java object has some members whose types are external java classes in external libraries). Hence, I wonder there is a tool can get the information from object's .class file and generate the Avro schema for that object (like Gson use object's .class information to convert certain object to JSON string).

Upvotes: 35

Views: 70187

Answers (4)

Joachim Lous
Joachim Lous

Reputation: 1541

AvroMapper() from jackson-dataformat-avro 2.5+ makes it really easy:

AvroMapper().schemaFor(MyClass.class.java).avroSchema

It is a subclass of jackson ObjectMapper, so all the regular jackson configs and extensions apply, e.g:

val mapper = AvroMapper().apply {registerModule(JavaTimeModule())}

Upvotes: 0

abasar
abasar

Reputation: 1668

** Example**

Pojo class 

public class ExportData implements Serializable {
    private String body;
    // ... getters and setters
}

Serialize

File file = new File(fileName);
DatumWriter<ExportData> writer = new ReflectDatumWriter<>(ExportData.class);
DataFileWriter<ExportData> dataFileWriter = new DataFileWriter<>(writer);
Schema schema = ReflectData.get().getSchema(ExportData.class);
dataFileWriter.create(schema, file);
for (Row row : resultSet) {
    String rec = row.getString(0);
    dataFileWriter.append(new ExportData(rec));
}
dataFileWriter.close();

Deserialize

File file = new File(avroFilePath);
DatumReader<ExportData> datumReader = new ReflectDatumReader<>(ExportData.class);
DataFileReader<ExportData> dataFileReader = new DataFileReader<>(file, datumReader);
ExportData record = null;
while (dataFileReader.hasNext()){
    record = dataFileReader.next(record);
    // process record
}

Upvotes: 5

charlb
charlb

Reputation: 1249

Here's how to Generate an Avro Schema from POJO definition

ObjectMapper mapper = new ObjectMapper(new AvroFactory());
AvroSchemaGenerator gen = new AvroSchemaGenerator();
mapper.acceptJsonFormatVisitor(RootType.class, gen);
AvroSchema schemaWrapper = gen.getGeneratedSchema();
org.apache.avro.Schema avroSchema = schemaWrapper.getAvroSchema();
String asJson = avroSchema.toString(true);

Upvotes: 14

MoustafaAAtta
MoustafaAAtta

Reputation: 1081

Take a look at the Java reflection API.

Getting a schema looks like:

Schema schema = ReflectData.get().getSchema(T);

See the example from Doug on another question for a working example.

Credits of this answer belong to Sean Busby.

Upvotes: 45

Related Questions