CODEWITHSUNDEEP
apache-sparkapache-kafka
user2521784
user2521784

Reputation: 107

How do I set up a 3 node Kafka Cluster

I need to setup 3 node Kafka cluster with 30 topics connecting to number of source DBs. I am not sure an exact number of messages to be processed but I guess there will be millions per day.

The retention of Kafka Topics and data is to be set at 30 days. I should able to get back to 30 days earlier data in case needed.

Could you please advise me how I should proceed.

Many Thanks

Upvotes: 0

Views: 4160

Answers (1)

Faisal Ahmed Siddiqui
Faisal Ahmed Siddiqui

Reputation: 707

SETUP: You can setup multi broker cluster by following instructions in given documentation. https://kafka.apache.org/quickstart#quickstart_multibroker

In 3 node cluster, you should configure 3 brokers and 3 zookeeper nodes and replication factor 3 for your topics. Depending on your requirement/use case, you can have 3 or more partitions for each topic.

RETENTION: You can define retention policy using log.retention.hours https://kafka.apache.org/documentation/#brokerconfigs

You should gather more information on how your data would be produced and consumed. Depending on your processing logic at consumer side, consuming one million messages per partition shouldn't be big deal. Though, You can add more partitions for parallel processing and increased throughput.

You should be specific in your question, but i tried and hope this helps!

Upvotes: 2

Related Questions