Rate Limit per user for sending notification like 1 notification in X seconds

Question

I have a use case where there is a stream of messages in Queue and I want to send this message to the user using a consumer job with these limitations

• There a limit like I can send only 1 notification in x seconds for example like 1 notification in every 10 seconds
• These messages should be in the same order when received by the user
• Consumer job should not block a thread for x seconds while delaying notification for user

For example: Stream has 100 messages from different users, a user y has got 5 messages in stream, and the notification limit is 1 notification every 10 seconds, now this consumer job should read from the stream and send these 5 notifications to the user in every 10 seconds in the same order received in a stream

Answer 1

Dump the messages into a database as they're consumed, and include the timestamp of the event. This will preserve ordering.

Otherwise, each user ID would map to a unique partition in Kafka and if you have millions of users, this will overwhelm the Kafka brokers... Alternatively, if two user's data are in the same partition, then you send notification for only the "second user", you should commit offsets to not process that data again, then you lose data from the "earlier user" without seeking backwards and potentially processing the "second user" data again.

Use a Timer in a background thread to query your database to count the events for each user every 10 seconds, if any exist, send the notification(s) and drop those records from your collection