CODEWITHSUNDEEP
apache-kafkaapache-kafka-connects3-kafka-connector
scriptkidie
scriptkidie

Reputation: 11

Kafka S3 sink connector failing with 22 topics

I am trying to use Kafka S3 sink connector to push the data from 22 topics to S3 bucket. While doing this, I am getting an error saying

`

ERROR [prod-partnerbilling-sink-v3|task-2] WorkerSinkTask{id=prod-partnerbilling-sink-v3-2} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:193)
org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception.
    at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:609)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:329)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:232)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:201)
    at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:186)
    at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:241)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.OutOfMemoryError: Java heap space

`

But when I use the same connector config for 2 batches of 11 topics each out of the 22 topics, it works fine.

I want to know the root cause of this error.

Upvotes: 0

Views: 540

Answers (1)

scriptkidie
scriptkidie

Reputation: 11

The issue was resolved after I lowered my s3.part.size from 50MB to 25MB.

Upvotes: 0

Related Questions