CODEWITHSUNDEEP
apache-kafka
Shubhangi_M
Shubhangi_M

Reputation: 55

How to stored the (Kafka) logs in to Database(Cassandra or Hive)?

How to stored the logs which is from producer to consumer into Database I follow below step after Download the 0.9.0.0 release and un-tar it.

  1. $ bin/zookeeper-server-start.sh config/zookeeper.properties &

  2. $ bin/kafka-server-start.sh config/server.properties

  3. $ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic fast-messages

$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic summary-markers

  1. $ bin/kafka-topics.sh --list --zookeeper localhost:2181

  2. $ cd .. $ mvn package

  3. Run the producer

  4. Start the consumer

Upvotes: 1

Views: 566

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191733

Download the 0.9.0.0 release and un-tar it.

That is a very old version of Kafka... All the steps you've used work the exact same on newer versions, and will likely be more stable.

Kafka doesn't store its data (logs) in a database.

If you want to send data that is produced to Kafka into another system, then the included tool (as of Kafka 0.10) is called Kafka Connect, and you can find Connectors that already exist for HDFS/Hive as well as Cassandra.

But you're welcome to write any Consumer to write to these systems on your own, Kafka just doesn't bundle this out of the box.

Upvotes: 2

Related Questions