CODEWITHSUNDEEP
apache-kafkaapache-kafka-connect
TheRealJimShady
TheRealJimShady

Reputation: 4303

Run kafka connect distributed mode on many nodes

I'm resiliency testing a kafka connector and I'd like to kill off a worker while it's running, thus killing the connector instance. The easiest way is probably going to be to force distributed mode to run over more than one node, then just kill the worker process on that node (right?). How can I make Kafka connect spawn workers on more than just the node it's started on? Is this something which is defined in worker config?

Upvotes: 2

Views: 3158

Answers (2)

TheRealJimShady
TheRealJimShady

Reputation: 4303

So in the end what I did was:

  • Copied all the jars I needed for Kafka Connect distributed mode to the two nodes I wanted to run it on (in HDP 2.5.3 you only get those jars on one node).
  • On both nodes, I ran the start script with properties file pointing to my jars.
  • Using the REST interface I posted my connector with a task, and I could see that one worker had the connector instance and another had its task.
  • I killed off the task worker node (using ps -ef | grep connect), and saw that it had respawned on the remaining node.
  • I reset the test and tried killing off the connector instance node, and to my amazement, the connector instance restarted on the other node.

In summary of my resiliency testing, Kafka Connect seems to be like playing whack-a-mole; you can kill off tasks or connectors wherever they are, and they will just respawn somewhere else.

Upvotes: 1

Robin Moffatt
Robin Moffatt

Reputation: 32090

Yes, handling failures and automatically restarting workload is exactly what Kafka Connect can do. You run it as a cluster, typically one worker per node. Each worker then runs one or many tasks, and this is managed by Connect. If a worker dies, all the tasks that it was running are restarted on other available workers, in a load balanced manner. Check out the architecture reference for more information.

To define workers as being within a cluster, assign them the same group.id. See the config docs for more info.

Upvotes: 2

Related Questions