CODEWITHSUNDEEP
apache-kafkaapache-kafka-connectconfluent-platform
Aloha Akbar
Aloha Akbar

Reputation: 33

org.apache.kafka.common.config.ConfigException: Unknown configuration 'errors.deadletterqueue.topic.name'

I'm using Kafka s3 Connector to push the json files into S3 bucket but I was not able to make my connector up and running. I'm using confluent 5.0 beta30 version.

Here is my connector configuration.

{
   "name": "custdb-s3-connector",
"config": {
    "connector.class": "io.confluent.connect.s3.S3SinkConnector",
    "key.converter":"org.apache.kafka.connect.json.JsonConverter",
    "tasks.max": "1",
    "topics": "CUST_ORDERS_ENRICHED",
    "s3.region": "us-west-2",
    "s3.bucket.name": "asif-datapipeline-demo",
    "s3.part.size": "5242880",
    "flush.size": "3",
    "storage.class": "io.confluent.connect.s3.storage.S3Storage",
    "format.class": "io.confluent.connect.s3.format.json.JsonFormat",
    "key.converter.schemas.enable":"false",
    "value.converter":"org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable":"false",
    "partition.field.name": "CUSTOMER_NUM",
    "schema.generator.class": "io.confluent.connect.storage.hive.schema.DefaultSchemaGenerator",
    "partitioner.class": "io.confluent.connect.storage.partitioner.DefaultPartitioner",
    "schema.compatibility": "NONE"
   }
}

In the connect logs we see the following error

connect            |  (org.apache.kafka.connect.runtime.errors.LogReporter$LogReporterConfig)
connect            | [2018-08-02 18:49:18,307] ERROR Failed to start task custdb-s3-connector-0 (org.apache.kafka.connect.runtime.Worker)
connect            | org.apache.kafka.common.config.ConfigException: Unknown configuration 'errors.deadletterqueue.topic.name'
connect            |    at org.apache.kafka.common.config.AbstractConfig.get(AbstractConfig.java:91)
connect            |    at org.apache.kafka.connect.runtime.ConnectorConfig$EnrichedConnectorConfig.get(ConnectorConfig.java:117)
connect            |    at org.apache.kafka.connect.runtime.ConnectorConfig.get(ConnectorConfig.java:162)
connect            |    at org.apache.kafka.common.config.AbstractConfig.getString(AbstractConfig.java:126)
connect            |    at org.apache.kafka.connect.runtime.Worker.sinkTaskReporters(Worker.java:531)
connect            |    at org.apache.kafka.connect.runtime.Worker.buildWorkerTask(Worker.java:508)
connect            |    at org.apache.kafka.connect.runtime.Worker.startTask(Worker.java:451)
connect            |    at org.apache.kafka.connect.runtime.distributed.DistributedHerder.startTask(DistributedHerder.java:873)
connect            |    at org.apache.kafka.connect.runtime.distributed.DistributedHerder.access$1600(DistributedHerder.java:111)
connect            |    at org.apache.kafka.connect.runtime.distributed.DistributedHerder$13.call(DistributedHerder.java:888)
connect            |    at org.apache.kafka.connect.runtime.distributed.DistributedHerder$13.call(DistributedHerder.java:884)
connect            |    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
connect            |    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
connect            |    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
connect            |    at java.lang.Thread.run(Thread.java:748)
connect            | [2018-08-02 18:49:18,293] INFO Instantiated connector mongodb-custdb-connector with version 0.9.0-SNAPSHOT o

Upvotes: 0

Views: 1447

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191671

See KAFKA-6981

You should use the 5.0 release of Confluent, rather than the Beta, where this issue has been resolved.

Upvotes: 0

Related Questions