CODEWITHSUNDEEP
javadesign-patternsarchitectureapache-kafkaconsumer
David Martin
David Martin

Reputation: 47

Kafka Consumer architecture design: java plugin or external client

I'm coding a solution for a data consumer to a kafka cluster (using dockers for kafka), but I still have to decide how to code it:

  1. In the native language Kafka is developed: java, in which the official site and git repo provides examples, https://github.com/apache/kafka/tree/trunk/examples/src/main/java/kafka/examples
  2. In another Client provided in another language, https://cwiki.apache.org/confluence/display/KAFKA/Clients, e.g. Node.js

The point is that I know how to code a solution in both languages, but I have to decide in which one having to take into account:

This is for a ubuntu server, on a full dockerized environment, I use Node.js as a core language for web services and connecting modules. I'm still not tunning the kafka client, but using the default options:

Java (using a KafkaConsumer)

import org.apache.kafka.clients.consumer.KafkaConsumer;
... 
props = ...
KafkaConsumer consumer = new KafkaConsumer<>(props);

Node.js (using kafka-node npm lib)

var kafka = require('kafka-node');
...
client = ....
var consumer = new kafka.Consumer( client, [{ topic: 'topicVehicle' }], { autoCommit: true });

I have to deal with the problem of maintaining the code at various layers in various languages (java, nodejs... probably python) and I'm wondering which is the more proper solution.

Upvotes: 0

Views: 739

Answers (1)

Marina
Marina

Reputation: 4064

My take is that you would not want to create your Kafka consumer as a Kafka plugin for a few reasons:

  • the main reason: creating standalone consumer allows you to scale it out and in based on the load, as needed, by adding or removing consumer instances for the same consumer group, letting Kafka re-distribute available partitions for processing among those consumer instances

other reasons:

  • plugin is something that usually extends functionality of whatever you are writing the plugin for - Kafka in this case
  • the biggest advantage of Kafka as a distributed message system is that you can have many independent consumers each doing their own thing, coming and going as they need to
  • creation and deployment of those consumers should not be coupled with the deployment of Kafka (or its plugins) - you should be able to add/remove consumers with no dependencies on Kafka itself

Now, if you are developing some new type of Connector or another consumer that is really intended to be more like a framework/tool that will be used by other consumers/producers - it would make sense to develop it as a plugin for Kafka

Upvotes: 2

Related Questions