CODEWITHSUNDEEP
apache-kafkaapache-kafka-connect
Aashish Verma
Aashish Verma

Reputation: 103

how to write kafka to kafka sink connector

i was trying for kafka mirror. And i found this open source [Comcast]: https://github.com/Comcast/MirrorTool-for-Kafka-Connect and it is successfully working when retrieving data form source kafka topic and writing it to destination kafka topic. now i need to sink from destination kafka topic and writing it to source kafka. How can i do that please suggest references.

i wrote kafkaSinkTask file. and inside put method i am able to get topics from destination kafka. so i don't know how to write this topics to source kfafa

    @Override
    public void put(Collection<SinkRecord> records) {
        if (records.isEmpty()) {
              return;
            }
            final SinkRecord first = records.iterator().next();
            final int recordsCount = records.size();
            logger.info(
                "Received {} records. First record kafka coordinates:({}-{}-{}).
                + "database...",
                recordsCount, first.topic(), first.kafkaPartition(), first.kafkaOffset()
            );  
    }

Upvotes: 1

Views: 3475

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191743

now i need to sink from destination kafka topic and writing it to source kafka

Can you not just flip the source and destination servers in your config file? Connect should generally consume from the remote cluster and produce to a local one (given two data centers or geographically separated networks), and store the consumed offsets at the destination in the Connect offsets topic, thus it being a source connector.

You could also try MirrorMaker 2.0 instead - https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0

Upvotes: 1

Related Questions