CODEWITHSUNDEEP
apache-kafkaapache-kafka-streams
acmoune
acmoune

Reputation: 3411

How to query a database from a Kafka processor?

I am trying to create an Account Management service using Kafka and Kafka Streams.

SignupRequest messages are placed on a signup-requests topic, and the first processor in the stream consuming that topic must check email uniqueness first, that is where the problems start, I am thinking on 2 possibilities but I am just a newbie ...

The first one would be to create a KTable on the accounts topic, so I can check email uniqueness with it. But I read that messages in a topic have a time to leave, after which they are deleted. So if an account with the checked email was created a long time ago greater than the configured time to leave, then it should not be present it the KTable, and my verification will be compromised.

The second option would be to query the database where the accounts are really persisted directly, but how to do async operations inside a kafka processor ? and is it a good practice ?

Upvotes: 0

Views: 262

Answers (1)

Robin Moffatt
Robin Moffatt

Reputation: 32050

I read that messages in a topic have a time to leave, after which they are deleted.

You can define retention based on time (or size of a topic), but can you also configure a topic as compacted. This is a special retention option which means that for every key, the latest message is always preserved—regardless of when it was received. Compacted topics are therefore great for topics that sit behind KTables.

Upvotes: 3

Related Questions