CODEWITHSUNDEEP
javascriptnode.jsapache-kafka
Sohail
Sohail

Reputation: 597

Database operations or http calls using kafka consumer

I am new to Apache Kafka. I have a few questions? if you produce topic using Kafka producer and the Kafka consumer received the message 

consumer.on('message', function (message) {
})

so for example, if I want to send data to some DB or to call in API to update my DB, Is it better to do all the codding inside consumer.on method

 consumer.on('message', function (message) {
    let data = await getPayload();//get payload
    //console.log(data,'data')
    //pause(topic,0); //pause until the prev request is completed
    request({
        url: process.env.API + '/some-api',
        method: 'POST',
        json: true,
        body: data
      }, function(err, response, body) {
        if (err) {
          console.error('got error', err);
          //resume(topic,0); resume when err/response recived 
        } else {
          // console.log('response', response);
          console.log('body', body);
          //resume(topic,0);
        }
    });
  })

Is there any alternative or recomended methods?

Upvotes: 0

Views: 222

Answers (1)

Liju John
Liju John

Reputation: 1876

While consuming from kafka and loading it to any sink like DB in your case , you need to be aware handling any failures, exactly one semantics etc. There is already a database sink connector available (link below), you can check if it suffice you requirements

kafka-connect-jdbc

Upvotes: 1

Related Questions