kafka_connect not finding its connectors

Question

I have a confluent kafka cluster running inside docker containers on EC2 machines as below

• 3 docker containers on separate EC2 instances for zookeepers and kafka-brokers
• 1 docker container on a separate EC2 instance for
  confluent_control_center
• 1 docker container on the same EC2 instance that the above control-center is running for kafka_connect

Below are the configurations for control_center and kafka_connect

CONTROL-CENTER

bootstrap.servers=<ec2_1:9092,ec2_2:9092,ec_3:9092>
zookeeper.connect=<ec2_1:2181,ec2_2:2181,ec_3:2181>
confluent.controlcenter.data.dir=/var/lib/confluent-control-center
confluent.monitoring.interceptor.topic.replication=1
confluent.controlcenter.internal.topics.replication=1
confluent.controlcenter.command.topic.replication=1
confluent.metrics.topic.replication=1
confluent.controlcenter.internal.topics.partitions=1
confluent.monitoring.interceptor.topic.partitions=1
confluent.controlcenter.config.dir=/etc/confluent-control-center
confluent.controlcenter.streams.num.stream.threads=2
confluent.controlcenter.replication.factor=1
confluent.controlcenter.connect.cluster=http://<kafka_connect_ec2>:8083

KAFKA-CONNECT

config.storage.topic=connect_config
log4j.root.loglevel=INFO
group.id=kafka-connect
bootstrap.servers=<ec2_1:9092,ec2_2:9092,ec_3:9092>
plugin.path=/usr/share/confluent-hub-components
key.converter=org.apache.kafka.connect.json.JsonConverter
offset.storage.topic=connect-offsets
internal.key.converter.schemas.enable=false
consumer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor
value.converter=org.apache.kafka.connect.json.JsonConverter
status.storage.topic=connect-status
producer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor
internal.value.converter.schemas.enable=false
rest.advertised.host.name=CONNECT
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter=org.apache.kafka.connect.json.JsonConverter

And in below snipped I am showing that I have gcs connector installed in the plugins directory

root@76763ee93675:/usr/share/confluent-hub-components# ls
confluentinc-kafka-connect-gcs

However, when I start the containers I can not see the gcs neither by reaching the kafka-connect rest api by using this

http://kafka_connect_ec2:8083/connectors

the above returns an empty list []

nor can I see the gcs connector in control_center

Some of the posts suggested to move the actual gcs jar file into the plugin directory, I tried that with no luck.

Another thing that I can't makes sense of, in the picture you can see that there are 2 connectors that control_center can "see"

• SchemaSourceConnector
• FileStreamSourceConnector

I am not sure where is the control_center finding those.

Can someone please advise what is being misconfigured here?

Thanks in advance.

Answer 1

In case if someone else stumbles upon an issue like this... What was happening was that the docker container where the kafka-connect was running did not have enough resources to load all the connectors, so it either would load some of the connectors and omit the rest or it would run out of resources and make the host unreachable. I wish the Kafka Connect Logs would reflect the fact that the connector loading was failing. Anyhow, since I am running my setup on an EC2 instance, all I had to do is to upgrade my instance size and this solved the issue. Thanks!

Answer 2

Control Center likely is not the issue, so no configuration would be needed there outside of the connector url.

You can lookup /connector-plugins of the Connect server to see the same things.

You need to ensure that the plugin.path is setup correctly to read from both /usr/share/java as well as /usr/share/confluent-hub-components to see both confluent hub installed ones as well as any preinstalled connectors.

Those two that you listed are part of the CLASSPATH variable, which Connect process also picks up

Note: According to the Confluent documentation, the GCS connector is already part of the Kafka Connect Docker images.