CODEWITHSUNDEEP
apache-kafkaapache-kafka-connectconfluent-schema-registry
Fares
Fares

Reputation: 650

Sink events of a same Kafka topic into multiple paths in GCS

I am using the Schema Registry with RecordNameStrategy naming policy so I have events with totally different avro schemas into the same Kafka topic.

I am doing that as I want to group logically related events that may have different data structures under a same topic to keep order for these data.

For instance: user_created event and user_mail_confirmed event might have different schemas but it's important to keep them into a same topic partition to guarantee order for consumers.

I am trying to sink these data, coming from a single topic, into GCS in multiple paths (one path for each schema)

Does someone know if the Confluent Kafka connect GCS Sink connector (or any other connector) provide us with that feature please ?

Upvotes: 0

Views: 274

Answers (1)

planky
planky

Reputation: 440

I haven't used GCS connector, but I suppose that this is not possible with Confluent connectors in general.

You should probably copy your source topic with different data structures to a new set of topics, where data have common data structure. This is possible with ksqlDB (check an example) or Kafka Streams application. Then, you can create connectors for these topics.

Alternatively, you can use RegexRouter transformation with a set of predicates based on the message headers.

Upvotes: 1

Related Questions