CODEWITHSUNDEEP
apache-kafkaapache-kafka-connect
c.mccafferty
c.mccafferty

Reputation: 31

Run Kafka and Kafka-connect on different servers?

I want to know if Kafka and Kafka-connect can run on different servers? So a connector would be started on server A and send data from a kafka topic on server B to HDFS or S3 etc. Thanks

Upvotes: 3

Views: 2936

Answers (2)

Prasad
Prasad

Reputation: 529

Yes, you can do it.

I have my set of kafka servers and kafka connect applications are running in different machines and writing data in hdfs. you have to mention list of brokers in bootstrap.servers under worker properties file (config/connect-distributed.properties or config/connect-standalone.properties) instead of localhost:9092

Upvotes: 1

Robin Moffatt
Robin Moffatt

Reputation: 32100

Yes, and for Production deployments this is typically recommended for resource reasons. Generally you'd deploy a cluster of Kafka Brokers (3+ for HA), and then a cluster of Kafka Cluster workers (as many as needed for throughput capacity / resilience) -- all on separate nodes.

For more details, see the Confluent Enterprise Reference Architecture.

Upvotes: 3

Related Questions