CODEWITHSUNDEEP
apache-kafkaapache-kafka-connect
Francesco
Francesco

Reputation: 129

Kafka Connect : "Task already exists in this worker"

I see the status of my connector and i have this exception

org.apache.kafka.connect.errors.ConnectException: Task already exists in this worker: *name of the connector*
        at org.apache.kafka.connect.runtime.Worker.startTask(Worker.java:365)
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder.startTask(DistributedHerder.java:857)
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder.access$1600(DistributedHerder.java:109)
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder$13.call(DistributedHerder.java:871)
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder$13.call(DistributedHerder.java:867)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

When i execute curl -s ip:8081/connectors/name of connector/tasks command for to view list of tasks of the connector, i have this result

{"error_code":404,"message":"HTTP 404 Not Found"}

I have no idea what it is. Thanks

Upvotes: 5

Views: 2362

Answers (1)

Anders Eriksson
Anders Eriksson

Reputation: 603

We are seeing this problem happening a couple of times in a single node distributed Connect cluster (latest version 2.0.0 of Kafka). We are seeing this with the Confluent JDBC Sink configured with 1 task.

Looking into the source - and the stacktrace - it seems the org.apache.kafka.connect.runtime.Worker and the org.apache.kafka.connect.runtime.distributed.DistributedHerder do not load task state in the same way. Somehow the Worker already have the task 0 but the DistributedHerder wants to add it (again).

A restart of the task as well as a restart of the connector did not help us. Instead a restart of the Connect process resolved it in our case.

Upvotes: 2

Related Questions