CODEWITHSUNDEEP
apache-kafkaapache-kafka-streamsapache-kafka-connectksqldb
Adarshlal
Adarshlal

Reputation: 77

Querying MySQL tables using Apache Kafka

I am trying to use Kafka Streams for achieving a use-case. I have two tables in MySQL - User and Account. And I am getting events from MySQL into Kafka using a Kafka MySQL connector.

I need to get all user-IDs within an account from within Kafka itself. So I was planning to use KStream on MySQL output topic, process it to form an output and publish it to a topic with Key as the account-id and value as the userIds separated by comma (,). Then I can use interactive query to get all userIds using account id, with the get() method of ReadOnlyKeyValueStore class. Is this the right way to do this? Is there a better way? Can KSQL be used here?

Upvotes: 0

Views: 1079

Answers (2)

capkutay
capkutay

Reputation: 183

I would take a look at striim if you want built in CDC and interactive continuous SQL queries on the streaming data in one UI. More info here:

http://www.striim.com/blog/2017/08/making-apache-kafka-processing-preparation-kafka/

Upvotes: 0

Robin Moffatt
Robin Moffatt

Reputation: 32050

You can use Kafka Connect to stream data in from MySQL, e.g. using Debezium. From here you can use KStreams, or KSQL, to transform the data, including re-keying which I think is what you're looking to do here, as well as join it to other streams. If you ingest the data from MySQL into a topic with log compaction set then you are guaranteed to always have the latest value for every key in the topic.

Upvotes: 4

Related Questions