CODEWITHSUNDEEP
apache-kafkakafka-consumer-apiapache-kafka-streamskafka-producer-api
Owen
Owen

Reputation: 161

How to split one Kafka topic into multiple smaller Kafka topics?

I have one main Kafka topic that receives some time series data. I need to take each value that comes into that topic, copy it, and send it to one of many separate topics based on a value in its key. Since it's time series data, each entry has a timestamp in its value. How can I accomplish this split while ensuring that the values pushed into the separate topics don't get out of order in respect to their timestamp?

Upvotes: 2

Views: 3776

Answers (2)

Rene B.
Rene B.

Reputation: 7424

You could use KSQL and create new topics with the SQL query:

CREATE STREAM pageviews(
    viewtime BIGINT KEY,
    userid VARCHAR,
    pageid VARCHAR
  ) WITH (
    KAFKA_TOPIC='pageviews',
    VALUE_FORMAT='DELIMITED',
    PARTITIONS=4,
    REPLICAS=3
  );

https://docs.ksqldb.io/en/latest/developer-guide/create-a-stream/

Or as @Matthias J. Sax mentioned with KStreams: https://kafka.apache.org/23/javadoc/org/apache/kafka/streams/kstream/KStream.html

Upvotes: 0

user1065569
user1065569

Reputation:

True. With kafkaStreams you can continuously read from any topic in the broker, process the data with a conditional (based in your case in the id) and write back to the broker in any other output topics. Optionally, to check results in them you can subscribe to these output topics from any other listener. Is easy and fast.

Upvotes: 2

Related Questions