CODEWITHSUNDEEP
apache-kafkaapache-kafka-connectconfluent-platform
frollo
frollo

Reputation: 1374

Connector config contains no connector type

I'm trying to use JDBC Connector to connect to a PostgreSQL database on my cluster (the database is not directly managed by the cluster).

I've been calling the Kafka Connect with the following command:

connect-standalone.sh worker.properties jdbc-connector.properties

This is the content of the worker.propertiesfile:

class=io.confluent.connect.jdbc.JdbcSourceConnector
name=test-postgres-1
tasks.max=1

internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false

offset.storage.file.filename=/home/user/offest
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter=org.apache.kafka.connect.json.JsonConverter

connection.url=jdbc:postgresql://database-server.url:port/database?user=user&password=password

And this are the content of the jdbc-connector.properties:

mode=incrementing
incrementing.column.name=id
topic.prefix=test-postgres-jdbc-

When I try to launch the connector with the above command it crashes with the following error: 

[2018-04-16 11:39:08,164] ERROR Failed to create job for jdbc.properties (org.apache.kafka.connect.cli.ConnectStandalone:88)
[2018-04-16 11:39:08,166] ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:99)
java.util.concurrent.ExecutionException: org.apache.kafka.connect.runtime.rest.errors.BadRequestException: Connector config {mode=incrementing, incrementing.column.name=pdv, topic.prefix=test-postgres-jdbc-} contains no connector type
    at org.apache.kafka.connect.util.ConvertingFutureCallback.result(ConvertingFutureCallback.java:80)
    at org.apache.kafka.connect.util.ConvertingFutureCallback.get(ConvertingFutureCallback.java:67)
    at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:96)
Caused by: org.apache.kafka.connect.runtime.rest.errors.BadRequestException: Connector config {mode=incrementing, incrementing.column.name=id, topic.prefix=test-postgres-jdbc-} contains no connector type
    at org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:233)
    at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.putConnectorConfig(StandaloneHerder.java:158)
    at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:93)

After noting that the connector causing the error displayed only informations from jdbc-connector.properties I've tried merging the two files together, but then the command terminates abruptly (without creating a topic or an offset file) with the following output:

[SLF4J infos...]
[2018-04-16 11:48:54,620] INFO Usage: ConnectStandalone worker.properties connector1.properties [connector2.properties ...] (org.apache.kafka.connect.cli.ConnectStandalone:59)

Upvotes: 5

Views: 7179

Answers (2)

goodfella
goodfella

Reputation: 194

For me I was using source properties as a .JSON file later I changed it to a .property file and worked.If you already using .property file check the below things also

  1. Ensure plugin.path is set correct in the connect-standalone/connect-distribured property file. If you have multiple folder put them comma separated.
  2. Ensure java version compatibility
  3. Ensure 'kafka connect' and 'connector plugin' compatibility.

Upvotes: 0

Robin Moffatt
Robin Moffatt

Reputation: 32090

You need to have most of those properties in the jdbc-connector.properties, not the worker.properties. See https://docs.confluent.io/current/connect/connect-jdbc/docs/source_config_options.html for a full list of config options that go in the connector configuration (jdbc-connector.properties in your example).

Try this:

  • worker.properties:

    internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false

offset.storage.file.filename=/home/user/offest
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter=org.apache.kafka.connect.json.JsonConverter

  • jdbc-connector.properties:

    class=io.confluent.connect.jdbc.JdbcSourceConnector
name=test-postgres-1
tasks.max=1

mode=incrementing
incrementing.column.name=id
topic.prefix=test-postgres-jdbc-

connection.url=jdbc:postgresql://database-server.url:port/database?user=user&password=password

You can see some more examples with Kafka Connect here:

Upvotes: 3

Related Questions