CODEWITHSUNDEEP
apache-kafkapluginsapache-kafka-connect
Ameya
Ameya

Reputation: 1960

Is there a documentation/blog/example on creating Kafka sink or source plugins?

We are planning to create our own repo of connector (sink or source plugins) for Apache Kafka like one here

We tried to search for the documentation or help on how to create a plugin jar for Kafka.

There is no mention of developing a plugin in the official documentation from apache Kafka.

Any help or pointer will be helpful, can share it back with the open community once developed.

Upvotes: 0

Views: 52

Answers (1)

Ran Lupovich
Ran Lupovich

Reputation: 1821

Here is guide on How to Build a Connector

As Well here is a Connector Developer Guide

Developing a connector only requires implementing two interfaces, the Connector and Task

Refer to the example source code for full examples for simple example

Once you’ve developed and tested your connector, you must package it so that it can be easily installed into Kafka Connect installations. The two techniques described here both work with Kafka Connect’s plugin path mechanism.

If you plan to package your connector and distribute it for others to use, you are obligated to properly license and copyright your own code and to adhere to the licensing and copyrights of all libraries your code uses and that you include in your distribution.

Creating an Archive The most common approach to packaging a connector is to create a tarball or ZIP archive. The archive should contain a single directory whose name will be unique relative to other connector implementations, and will therefore often include the connector’s name and version. All of the JAR files and other resource files needed by the connector, including third party libraries, should be placed within that top-level directory. Note, however, that the archive should never include the Kafka Connect API or runtime libraries.

To install the connector, a user simply unpacks the archive into the desired location. Having the name of the archive’s top-level directory be unique makes it easier to unpack the archive without overwriting existing files. It also makes it easy to place this directory on Installing Connect Plugins or for older Kafka Connect installations to add the JARs to the CLASSPATH.

Creating an Uber JAR An alternative approach is to create an uber JAR that contains all of the connector’s JAR files and other resource files. No directory internal structure is necessary.

To install, a user simply places the connector’s uber JAR into one of the directories listed in Installing Connect Plugins.

Upvotes: 2

Related Questions