CODEWITHSUNDEEP
amazon-s3apache-kafkaapache-kafka-connectconfluent-platform
user8072466
user8072466

Reputation: 83

kafka connect failes to write to S3 with - ERROR Multipart upload

We have Kafka connect running on Kube pods. We are seeing the following errors in the worker logs. We tried restarting the pods. 

[2020-04-02 14:40:13,237] WARN Aborting multi-part upload with id '0vbfuRZCRIkc431LJN.hyaPuo3ZQzuAsfTrMSdBE_Q9.sZP8g-' (io.confluent.connect.s3.storage.S3OutputStream)
[2020-04-02 14:40:13,274] ERROR Multipart upload failed to complete for bucket 'my_bucket' key '/myfile':  (io.confluent.connect.s3.TopicPartitionWriter)
org.apache.kafka.connect.errors.DataException: Multipart upload failed to complete.
    at io.confluent.connect.s3.storage.S3OutputStream.commit(S3OutputStream.java:159)
    at io.confluent.connect.s3.format.avro.AvroRecordWriterProvider$1.commit(AvroRecordWriterProvider.java:96)
    at io.confluent.connect.s3.TopicPartitionWriter.commitFile(TopicPartitionWriter.java:503)
    at io.confluent.connect.s3.TopicPartitionWriter.commitFiles(TopicPartitionWriter.java:483)
    at io.confluent.connect.s3.TopicPartitionWriter.commitOnTimeIfNoData(TopicPartitionWriter.java:294)
    at io.confluent.connect.s3.TopicPartitionWriter.write(TopicPartitionWriter.java:184)
    at io.confluent.connect.s3.S3SinkTask.put(S3SinkTask.java:193)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:538)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:321)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:224)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:192)
    at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:177)
    at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:227)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Part upload failed:
    at io.confluent.connect.s3.storage.S3OutputStream.uploadPart(S3OutputStream.java:136)
    at io.confluent.connect.s3.storage.S3OutputStream.commit(S3OutputStream.java:153)
    ... 17 more
[2020-04-02 14:40:13,275] INFO Committing files after waiting for rotateIntervalMs time but less than flush.size records available. (io.confluent.connect.s3.TopicPartitionWriter)

Upvotes: 2

Views: 2810

Answers (3)

chris-vest
chris-vest

Reputation: 31

In my case this was two things:

  1. KMS encryption had been enabled on the bucket so the policy attached to the Connector's user needed access to the KMS key in question; Connector needed the following KMS permissions:
  "Action": [
    "kms:Encrypt",
    "kms:Decrypt",
    "kms:ReEncrypt*",
    "kms:GenerateDataKey",
    "kms:GenerateDataKey*",
    "kms:DescribeKey"
  ],
  1. An SCP was enabled on the account which disabled s3:PutObject if "s3:x-amz-server-side-encryption": "false" - removing this solved the issue

This document turned out to be useful: https://aws.amazon.com/premiumsupport/knowledge-center/s3-troubleshoot-403/

Upvotes: 1

user8072466
user8072466

Reputation: 83

The root cause was because of missing appropriate KMS key policy.

Upvotes: 0

Robin Moffatt
Robin Moffatt

Reputation: 32090

DataException: Multipart upload failed to complete.
java.io.IOException: Part upload failed

Looks like the connection to S3 was interrupted. Restart the connector, and check your network connectivity.

Upvotes: 0

Related Questions