CODEWITHSUNDEEP
apache-kafkakcat
Anthony
Anthony

Reputation: 35988

How to load data from json file into kafka

I have a mydata.json file that is like below:

{
  "student_id": "Student 01",
  "grades": [
    {
      "english": "A",
      "semester": 1
    },
    {
      "math": "B",
      "semester": 2
    }
  ],
  "attendance": [
    {
      "present": false,
      "timestamp": "2021-11-02"
    },
    {
      "present": true,
      "timestamp": "2021-09-02"
    }
  ]
}
{
  "student_id": "Student 02",
  "grades": [
    {
      "english": "C",
      "semester": 2
    },
    {
      "math": "A",
      "semester": 1
    }
  ],
  "attendance": [
    {
      "present": true,
      "timestamp": "2021-08-02"
    },
    {
      "present": false,
      "timestamp": "2021-07-02"
    }
  ]
}

The above sample file contains two elements: Student 01 and Student 02. I am trying to load this into a kafka topic student_data such that each student (i.e. Student 01 and Student 02) becomes a separate message.

I tried using kafkacat command below but it is loading the entire file into a single message on the topic.

kcat -P -b localhost:29092 -t student_data mydata.json

Is there a way to add this to the topic so that each entry is a separate message? ...without breaking each message into a separate file.

Upvotes: 0

Views: 2727

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191874

What you've shown is not a valid JSON file. Also, the CLI tools only parse line-delimited formats

If you flattened the file into JSONlines format, use redirection

kcat -P  ... < data.jsonl

https://jsonlines.org/examples/

The same applies for kafka-console-producer

Upvotes: 4

Related Questions