CODEWITHSUNDEEP
jsonapache-kafkaavroconfluent-schema-registry
Philipp
Philipp

Reputation: 420

Convert Json to Avro schema

Problem

I want to convert a normal Json file like below to an avro schema to make it work in apache kafka with the confluent schema registry.

Example

Input (Json)

[
  {
    "name": "Robin Hood",
    "department": "",
    "manager": "",
    "salary": 200
  },
  {
    "name": "Arsene Wenger",
    "department": "Bar",
    "manager": "Friar Tuck",
    "salary": 50
  },
  {
    "name": "Friar Tuck",
    "department": "Foo",
    "manager": "Robin Hood",
    "salary": 100
  },
  {
    "name": "Little John",
    "department": "Foo",
    "manager": "Robin Hood",
    "salary": 100
  },
  {
    "name": "Sam Allardyce",
    "department": "",
    "manager": "",
    "salary": 250
  },
  {
    "name": "Dimi Berbatov",
    "department": "Foo",
    "manager": "Little John",
    "salary": 50
  }
]

Output (Avro schema) 

{
  "name": "MyClass",
  "type": "array",
  "namespace": "com.acme.avro",
  "items": {
    "name": "MyClass_record",
    "type": "record",
    "fields": [
      {
        "name": "name",
        "type": "string"
      },
      {
        "name": "department",
        "type": "string"
      },
      {
        "name": "manager",
        "type": "string"
      },
      {
        "name": "salary",
        "type": "int"
      }
    ]
  }
}

A Json Schema as input would be good as well.

This question was asked a while ago but hast no good answer.

There is a website which does this but I want a library or cli.

Thanks!

Upvotes: 2

Views: 8118

Answers (1)

Philipp
Philipp

Reputation: 420

Turns out this is possible with avro4s. It is a Scala librarie, I foud nothing in java.

Here is a simple example on how to use it. I have events which contain properties. 

package example

import com.sksamuel.avro4s.AvroSchema
import com.sksamuel.avro4s.json.JsonToAvroConverter

object Main extends App {

  case class Propertie(name: String, value: String)
  case class Event(name: String, properties: Seq[Propertie])

  val schema = AvroSchema[Event]

  val converter = new JsonToAvroConverter("com.example.kafkaorch")
  val string =
    """{
      |  "AvroEvent": {
      |    "name": "order-created",
      |    "AvroPropertie": {
      |      "name": "",
      |      "type": "",
      |      "value":""
      |    }
      |  }
      |}""".stripMargin

  print(converter.convert("test", string).toString(true))

}

The result should be this: 

{
  "type" : "record",
  "name" : "test",
  "namespace" : "com.example.kafkaorch",
  "fields" : [ {
    "name" : "AvroEvent",
    "type" : {
      "type" : "record",
      "name" : "AvroEvent",
      "fields" : [ {
        "name" : "name",
        "type" : "string"
      }, {
        "name" : "AvroPropertie",
        "type" : {
          "type" : "record",
          "name" : "AvroPropertie",
          "fields" : [ {
            "name" : "name",
            "type" : "string"
          }, {
            "name" : "type",
            "type" : "string"
          }, {
            "name" : "value",
            "type" : "string"
          } ]
        }
      } ]
    }
  } ]
}

Upvotes: 0

Related Questions