CODEWITHSUNDEEP
multithreadingqueueapache-kafka
Sabarish Sathasivan
Sabarish Sathasivan

Reputation: 1276

Worker Queue option in Kafka

We are developing an application , which will receive time series sensor data as byte array from a set of devices via UDP. This data needs to be parsed and stored in a Cassandra Database...

We were using RabbitMQ as the message broker and using the Work Queues based consumers to parse the data and push it in to cassandra... Because of increasing traffic, we are concerned about RabbitMQ perfomance and are planning to move to Kafka... Our understanding is that the same can be implemented using consumer group in kafka .. is our understanding correct

Upvotes: 2

Views: 987

Answers (1)

devsprint
devsprint

Reputation: 671

With Apache Kafka, you can scale a topic relatively easier. In order to be able to process more data in same time you'll need:

  • Having multiple consumers in same consumer group, you'll be able to consume multiple messages in same time. You are limited to the number of partitions of a topic.
  • Increase the number of partitions for a topic, and increase the number of consumers.
  • Increase the number of brokers, if you still to process more data.

I will approach the scalability in the order described above, but Kafka can handle a lot. In a setup with 2 brokers, 4 partitions per topic and 2 consumers (each consumer use one thread per partition), consumer decode json to java object, enrich and store to Cassandra, it can handle 30k/s (data is batched in batch of 200 insert statements).

Upvotes: 1

Related Questions