CODEWITHSUNDEEP
cassandraapache-kafka
EugeneMi
EugeneMi

Reputation: 3575

Streaming data from Kafka into Cassandra in real time

What's the best way to write date from Kafka into Cassandra? I would expect it to be a solved problem, but there doesn't seem to be a standard adapter. A lot of people seem to be using Storm to read from Kafka and then write to Cassandra, but storm seems like somewhat of an overkill for simple ETL operations.

Upvotes: 9

Views: 5091

Answers (2)

Aravind Yarram
Aravind Yarram

Reputation: 80194

Recent release of Kafka came with the connector concept to support source and sinks as first class concepts in the design. With this, you do not need any streaming framework for moving data in/out of Kafka. Here is the Cassandra connector for Kafka that you can use: https://github.com/tuplejump/kafka-connect-cassandra

Upvotes: 1

viktortnk
viktortnk

Reputation: 2757

We are heavily using Kafka and Cassandra through Storm

We rely on Storm because:

  • there are usually a lot of distributed processing (inter-node) steps before result of original message hit Cassandra (Storm bolt topologies)

  • We don't need to maintain consumer state of Kafka (offset) ourselves - Storm-Kafka connector is doing it for us when all products of original message is acked within Storm

  • Message processing is distributed across nodes with Storm natively

Otherwise if it is a very simple case, you might effectively read messages from Kafka and write result to Cassandra without help of Storm

Upvotes: 5

Related Questions